Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems

With the current growth of the proposed contextual recommending algorithms, evaluating them becomes more critical. Researchers of recommender systems have expressed concerns that the evaluation quality cannot be properly judged. We carried out meta-analyses of the evaluation methods and metrics of 67 studies related to context-aware scholarly recommender systems, from the years 2000 to 2014. The analysis of variance results shows that offline evaluation methods are more commonly used compared to online and user studies, with the maximum rate of success. It also reveals the popularity order of accuracy metrics (31%) including “Recall, Precision, F-Measure”, “Mean Absolute Error, and Questionnaire studies, Reliability, Accessibility, Feasibility, Usability, Applicability and Performance”. By using factor analysis, 28 different evaluation metrics were classified into eight groups. The results of analysis have shown the difference in evaluation methods in applying different groups of metrics. This study highlights the importance of how an evaluation method should be adequately designed and implemented. Additionally, a few recommendations for future investigations on recommending evaluation are proposed.