Topics of Interest

We invite the submission of papers reporting original research, studies, advances, experiences, or work in progress in the scope of recommender system utility evaluation. The topics the workshop seeks to address include–though need not be limited to–the following:

  • Recommendation quality dimensions
    • Effective accuracy, ranking quality
    • Novelty, diversity, unexpectedness, serendipity
    • Utility, gain, cost, risk, benefit
    • Robustness, confidence, coverage, ease of use, persuasiveness, etc.
  • Matching metrics to tasks, needs, and goals
    • User satisfaction, user perception, human factors
    • Business-oriented evaluation
    • Multiple objective optimization, user engagement
    • Quality of service, quality of experience
  • Evaluation methodology and experimental design
    • Definition and evaluation of new metrics, studies of existing ones
    • Adaptation of methodologies from related fields: IR, Machine Learning, HCI, etc.
    • Evaluation theory
  • Practical aspects of evaluation
    • Offline and online experimental approaches
    • Simulation-based evaluation
    • Datasets and benchmarks
    • Validation of metrics
    • Efficiency and scalability
    • Open evaluation platforms and infrastructures