An Interactive Relevance Feedback Interface for Evidence-Based Health Care

We design, implement and evaluate EpistAid, an interactive relevance feedback system to support physicians towards a more efficient citation screening process for Evidence Based Health Care (EBHC). The system combines a relevance feedback algorithm with an interactive interface inspired by Tinder-like swipe interaction. To evaluate its efficiency and effectiveness in the citation screening process we conducted a user study with real users (senior medicine students) using a large EBHC dataset (Epistemonikos), with around 400,000 documents. We compared two relevance feedback algorithms, Rocchio and BM25-based. The combination of Rocchio relevance feedback with the document visualization yielded the best recall and F-1 scores, which are the most important metrics for EBHC document screening. In terms of cognitive demand and effort, BM25 relevance feedback without visualization was perceived as needing more physical and cognitive effort. EpistAid has the potential of improving the process for answering clinical questions by reducing the time needed to classify documents, as well as promoting user interaction. Our results can inform the development of intelligent user interfaces for screening research articles in the clinical domain and beyond.

[1]  Philip S. Yu,et al.  Evidence-based medicine, the essential role of systematic reviews, and the need for automated text mining tools , 2010, IHI.

[2]  Carla E. Brodley,et al.  Deploying an interactive machine learning system in an evidence-based practice center: abstrackr , 2012, IHI '12.

[3]  Carla E. Brodley,et al.  Who Should Label What? Instance Allocation in Multiple Expert Active Learning , 2011, SDM.

[4]  Daniel Pérez,et al.  Epistemonikos: A Free, Relational, Collaborative, Multilingual Database of Health Evidence , 2013, MedInfo.

[5]  Sophia Ananiadou,et al.  Supporting systematic reviews using LDA-based document representations , 2015, Systematic Reviews.

[6]  Dina Demner-Fushman,et al.  Feature Engineering and a Proposed Decision-Support System for Systematic Reviewers of Medical Evidence , 2014, PloS one.

[7]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[8]  Carla E. Brodley,et al.  Modeling annotation time to reduce workload in comparative effectiveness reviews , 2010, IHI.

[9]  Alfred Kobsa.  User Modeling and User-Adapted Interaction , 2005, User Modeling and User-Adapted Interaction.

[10]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[11]  Jaakko Peltonen,et al.  Negative Relevance Feedback for Exploratory Search with Visual Interactive Intent Modeling , 2017, IUI.

[12]  Christoph Trattner,et al.  See what you want to see: visual user-driven approach for hybrid recommendation , 2014, IUI.

[13]  Ivania Donoso-Guzmán.  EpistAid: An Interactive Intelligent System for Evidence-based Health Care , 2017, IUI Companion.

[14]  Olga Vechtomova,et al.  Book Review: Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze , 2009, CL.

[15]  Halil Kilicoglu,et al.  Viewpoint Paper: Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence , 2009, J. Am. Medical Informatics Assoc..

[16]  Erik Duval,et al.  Visualizing recommendations to support exploration, transparency and controllability , 2013, IUI '13.

[17]  Juan Jose García Adeva,et al.  Automatic text classification to support systematic reviews in medicine , 2014, Expert Syst. Appl..

[18]  Clive E Adams,et al.  Systematic reviews: Work that needs to be done and not to be done , 2013, Journal of evidence-based medicine.

[19]  Wes McKinney,et al.  Data Structures for Statistical Computing in Python , 2010, SciPy.

[20]  Muin J. Khoury,et al.  GAPscreener: An automatic tool for screening human genetic association literature in PubMed using the support vector machine technique , 2008, BMC Bioinformatics.

[21]  Daniel Engel,et al.  A Survey of Dimension Reduction Methods for High-dimensional Data Analysis and Visualization , 2011, VLUDS.

[22]  Sophia Ananiadou,et al.  Reducing systematic review workload through certainty-based screening , 2014, J. Biomed. Informatics.

[23]  Alfred Kobsa,et al.  Inspectability and control in social recommenders , 2012, RecSys.

[24]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[25]  Arnab Nandi,et al.  Don't Just Swipe Left, Tell Me Why: Enhancing Gesture-based Feedback with Reason Bins , 2017, IUI.

[26]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[27]  Carla E. Brodley,et al.  Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining , 2012, Genetics in Medicine.

[28]  Eduardo E. Veas,et al.  Rank As You Go: User-Driven Exploration of Search Results , 2016, IUI.

[29]  Nathalie Japkowicz,et al.  Using Classifier Performance Visualization to Improve Collective Ranking Techniques for Biomedical Abstracts Classification , 2010, Canadian Conference on AI.

[30]  Denis Parra,et al.  Moodplay: Interactive Mood-based Music Discovery and Recommendation , 2016, UMAP.

[31]  Bart P. Knijnenburg,et al.  Each to his own: how different users call for different interaction methods in recommender systems , 2011, RecSys '11.

[32]  Shuang-Hong Yang,et al.  Dimensionality Reduction and Topic Modeling: From Latent Semantic Indexing to Latent Dirichlet Allocation and Beyond , 2012, Mining Text Data.

[33]  Aaron M. Cohen,et al.  An Effective General Purpose Approach for Automated Biomedical Document Classification , 2006, AMIA.

[34]  Borim Ryu,et al.  Combining relevancy and methodological quality into a single ranking for evidence-based medicine , 2012, Inf. Sci..

[35]  Tamara Munzner,et al.  Visualization Analysis and Design , 2014, A.K. Peters visualization series.

[36]  Dietmar Jannach,et al.  A systematic review and taxonomy of explanations in decision support and recommender systems , 2017, User Modeling and User-Adapted Interaction.

[37]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[38]  Simone R. S. Souza,et al.  The Use of Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews: A Replication Study , 2013, 2013 3rd International Workshop on Replication in Empirical Software Engineering Research.

[39]  Rosane Minghim,et al.  A visual analysis approach to validate the selection review of primary studies in systematic reviews , 2012, Inf. Softw. Technol..

[40]  Shogo Nishida,et al.  The relation between user intervention and user satisfaction for information recommendation , 2012, SAC '12.

[41]  D. Sackett,et al.  On the need for evidence-based medicine. , 1995, Journal of public health medicine.

[42]  Tari Turner,et al.  Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap , 2014, PLoS medicine.

[43]  G. Antes,et al.  Five Steps to Conducting a Systematic Review , 2003, Journal of the Royal Society of Medicine.

[44]  Carla E. Brodley,et al.  Semi-automated screening of biomedical citations for systematic reviews , 2010, BMC Bioinformatics.

[45]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[46]  HeerJeffrey,et al.  D3 Data-Driven Documents , 2011 .

[47]  Tobias Höllerer,et al.  TasteWeights: a visual interactive hybrid recommender system , 2012, RecSys.

[48]  Shuguang Han,et al.  Supporting exploratory people search: a study of factor transparency and user control , 2013, CIKM.

[49]  Stan Matwin,et al.  Classifying Biomedical Abstracts Using Committees of Classifiers and Collective Ranking Techniques , 2009, Canadian Conference on AI.

[50]  Sophia Ananiadou,et al.  Applications of text mining within systematic reviews , 2011, Research synthesis methods.

[51]  Brian E. Howard,et al.  SWIFT-Review: a text-mining workbench for systematic review , 2016, Systematic Reviews.

[52]  Pearl Brereton,et al.  A critical analysis of studies that address the use of text mining for citation screening in systematic reviews , 2016, EASE.

[53]  Gaël Varoquaux,et al.  The NumPy Array: A Structure for Efficient Numerical Computation , 2011, Computing in Science & Engineering.

[54]  Siddhartha Jonnalagadda,et al.  A new iterative method to reduce workload in systematic review process , 2013, Int. J. Comput. Biol. Drug Des..

[55]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[56]  P. Glasziou,et al.  Are systematic reviews up-to-date at the time of publication? , 2013, Systematic Reviews.

[57]  Dina Demner-Fushman,et al.  Towards Automating the Initial Screening Phase of a Systematic Review , 2010, MedInfo.

[58]  S. Ananiadou,et al.  Using text mining for study identification in systematic reviews: a systematic review of current approaches , 2015, Systematic Reviews.