Harmonization and Categorization of Metrics and Criteria for Evaluation of Recommender Systems in Healthcare From Dual Perspectives

Researchers' choice of metrics and criteria in evaluating recommender systems depends on what the researcher feels is popular among other researchers, or sometimes based on the objective of the research. There is no harmonized set of criteria and metrics that can be referenced when evaluating recommender systems in healthcare. In this article, a set of metrics and criteria are harmonized and categorized as a guide for evaluating recommender systems. By means of an online survey, the opinions of forty-four experienced researchers and other stakeholders from eight countries and four continents were sought on the relevance of identified metrics and criteria. Analysis of the results show speed and timeliness are at the top. Topping the list of criteria is the provision of information that will guide users to useful decisions. The result is presented from two logical perspectives. Four categories are then identified as a useful guide for evaluating recommender systems.

[1]  Fuguo Zhang,et al.  Timeliness in recommender systems , 2017, Expert Syst. Appl..

[2]  Yinsheng Li,et al.  A Recommender System for Healthcare Based on Human-Centric Modeling , 2016, 2016 IEEE 13th International Conference on e-Business Engineering (ICEBE).

[3]  Martin Wiesner,et al.  Health Recommender Systems: Concepts, Requirements, Technical Basics and Challenges , 2014, International journal of environmental research and public health.

[4]  João Manuel R. S. Tavares,et al.  Novel and powerful 3D adaptive crisp active contour method applied in the segmentation of CT lung images , 2017, Medical Image Anal..

[5]  Pattie Maes,et al.  Social information filtering: algorithms for automating “word of mouth” , 1995, CHI '95.

[6]  Pedro Pedrosa Rebouças Filho,et al.  New approach to detect and classify stroke in skull CT images via analysis of brain tissue densities , 2017, Comput. Methods Programs Biomed..

[7]  Joel J. P. C. Rodrigues,et al.  Nature-Inspired Algorithm for Training Multilayer Perceptron Networks in e-health Environments for High-Risk Pregnancy Care , 2017, Journal of Medical Systems.

[8]  Jun Zhou,et al.  An Industrial-Scale System for Heterogeneous Information Card Ranking in Alipay , 2018, DASFAA.

[9]  Franca Garzotto,et al.  Looking for "Good" Recommendations: A Comparative Evaluation of Recommender Systems , 2011, INTERACT.

[10]  Hendrik Drachsler,et al.  User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg , 2018, IEEE Transactions on Learning Technologies.

[11]  Aditya G. Parameswaran,et al.  Towards Visualization Recommendation Systems , 2016, SGMD.

[12]  Gabriel-Miro Muntean,et al.  A recommender system architecture for predictive telecom network management , 2015, IEEE Communications Magazine.

[13]  William Nick Street,et al.  Healthcare information systems: data mining methods in the creation of a clinical recommender system , 2011, Enterp. Inf. Syst..

[14]  Ferran Torrent-Fontbona,et al.  Adaptive basal insulin recommender system based on Kalman filter for type 1 diabetes , 2018, Expert Syst. Appl..

[15]  Saul Vargas,et al.  Rank and relevance in novelty and diversity metrics for recommender systems , 2011, RecSys '11.

[16]  Clayton R. Pereira,et al.  Automated recognition of lung diseases in CT images based on the optimum-path forest classifier , 2017, Neural Computing and Applications.

[17]  Félix Hernández-del-Olmo,et al.  Evaluation of recommender systems: A new approach , 2008, Expert Syst. Appl..

[18]  Stephen Russell,et al.  Applications of wavelet data reduction in a recommender system , 2008, Expert Syst. Appl..

[19]  João Manuel R. S. Tavares,et al.  Analysis of human tissue densities: A new approach to extract features from medical images , 2017, Pattern Recognit. Lett..

[20]  Alejandro Bellogín,et al.  Precision-oriented evaluation of recommender systems: an algorithmic comparison , 2011, RecSys '11.

[21]  Peter G. Raeth,et al.  A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender Systems , 2017, Int. J. ICT Res. Afr. Middle East.

[22]  Nicholas Jing Yuan,et al.  Relevance Meets Coverage , 2016, ACM Trans. Intell. Syst. Technol..

[23]  Franca Garzotto,et al.  User-Centric vs. System-Centric Evaluation of Recommender Systems , 2013, INTERACT.

[24]  Guy Shani,et al.  Evaluating Recommendation Systems , 2011, Recommender Systems Handbook.

[25]  Yung-Ming Li,et al.  TREPPS: A Trust-based Recommender System for Peer Production Services , 2009, Expert Syst. Appl..

[26]  Kenneth Y. Goldberg,et al.  Eigentaste 5.0: constant-time adaptability in a recommender system using item clustering , 2007, RecSys '07.

[27]  Pablo Castells,et al.  Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems , 2018, SIGIR.

[28]  Gang Lv,et al.  Research on recommender system based on ontology and genetic algorithm , 2016, Neurocomputing.

[29]  Barry Smyth,et al.  A classification-based review recommender , 2010, Knowl. Based Syst..

[30]  Keijo Haataja,et al.  Systematic Literature Review on Empirical Results and Practical Implementations of Healthcare Recommender Systems: Lessons Learned and a Novel Proposal , 2015, Int. J. Heal. Inf. Syst. Informatics.

[31]  V Viji Rajendran,et al.  Fuzzy ontology based recommender system with diversification mechanism , 2017, 2017 International Conference on Intelligent Computing and Control (I2C2).

[32]  Paolo Tomeo,et al.  Schema-summarization in linked-data-based feature selection for recommender systems , 2017, SAC.

[33]  Ivan V. Oseledets,et al.  Tensor methods and recommender systems , 2016, Wiley Interdiscip. Rev. Data Min. Knowl. Discov..

[34]  Silvana Aciar,et al.  Increasing effectiveness in e-commerce: recommendations applying intelligent agents , 2007 .

[35]  Beatrice Lazzerini,et al.  Using context history to personalize a resource recommender via a genetic algorithm , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[36]  Yi Zhang,et al.  Interaction and Personalization of Criteria in Recommender Systems , 2010, UMAP.

[37]  Deborah Estrin,et al.  OpenRec: A Modular Framework for Extensible and Adaptable Recommendation Algorithms , 2018, WSDM.

[38]  Neil J. Hurley,et al.  Collaborative recommendation: A robustness analysis , 2004, TOIT.

[39]  T. S. Cavalcante,et al.  3D Lung Fissure Segmentation in TC images based in Textures , 2016, IEEE Latin America Transactions.

[40]  Victor Hugo C. De Albuquerque,et al.  Health of Things Algorithms for Malignancy Level Classification of Lung Nodules , 2018, IEEE Access.

[41]  Katrien Verbert,et al.  Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities , 2016, Expert Syst. Appl..

[42]  Shara Shami A. Alves,et al.  A New Approach to Diagnose Parkinson's Disease Using a Structural Cooccurrence Matrix for a Similarity Analysis , 2018, Comput. Intell. Neurosci..

[43]  Liang He,et al.  Evaluating recommender systems , 2012, Seventh International Conference on Digital Information Management (ICDIM 2012).

[44]  S. C. Hui,et al.  Web content recommender system based on consumer behavior modeling , 2011, IEEE Transactions on Consumer Electronics.

[45]  Kevin MacG. Adams.  Adaptability, Flexibility, Modifiability and Scalability, and Robustness , 2015 .

[46]  Fátima N. S. de Medeiros,et al.  Lung disease detection using feature extraction and extreme learning machine , 2014 .

[47]  Mouzhi Ge,et al.  Beyond accuracy: evaluating recommender systems by coverage and serendipity , 2010, RecSys '10.

[48]  Li Chen,et al.  A user-centric evaluation framework for recommender systems , 2011, RecSys '11.

[49]  Qing Yang,et al.  Exploring timeliness for accurate recommendation in location-based social networks , 2018, Math. Found. Comput..

[50]  Sean M. McNee,et al.  Making recommendations better: an analytic model for human-recommender interaction , 2006, CHI Extended Abstracts.

[51]  Pekka Toivanen,et al.  Recommender Systems in Healthcare: Towards Practical implementation of Real-Time Recommendations to Meet the Needs of Modern Caregiving , 2018 .

[52]  Katrien Verbert,et al.  Recommender Systems for Health Informatics: State-of-the-Art and Future Perspectives , 2016, Machine Learning for Health Informatics.