Improving the conduct of systematic reviews: a process mining perspective.

OBJECTIVES To illustrate the use of process mining concepts, techniques, and tools to improve the systematic review process. STUDY DESIGN AND SETTING We simulated review activities and step-specific methods in the process for systematic reviews conducted by one research team over 1 year to generate an event log of activities, with start/end dates, reviewer assignment by expertise, and person-hours worked. Process mining techniques were applied to the event log to "discover" process models, which allowed visual display, animation, or replay of the simulated review activities. Summary statistics were calculated for person-time and timelines. We also analyzed the social networks of team interactions. RESULTS The 12 simulated reviews included an average of 3,831 titles/abstracts (range: 1,565-6,368) and 20 studies (6-42). The average review completion time was 463 days (range: 289-629) (881 person-hours [range: 243-1,752]). The average person-hours per activity were study selection 26%, data collection 24%, report preparation 23%, and meta-analysis 17%. Social network analyses showed the organizational interaction of team members, including how they worked together to complete review tasks and to hand over tasks upon completion. CONCLUSION Event log and process mining can be valuable tools for research teams interested in improving and modernizing the systematic review process.

[1]  C. Robertson,et al.  Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non‐randomised study designs applied in a health technology assessment setting , 2014, Research synthesis methods.

[2]  J. Hayden,et al.  Methods of a multi-faceted rapid knowledge synthesis project to inform the implementation of a new health service model: Collaborative Emergency Centres , 2015, Systematic Reviews.

[3]  Lisa Hartling,et al.  Single data extraction generated more errors than double data extraction in systematic reviews. , 2006, Journal of clinical epidemiology.

[4]  Marcos Sepúlveda,et al.  Clinical Processes and Its Data, What Can We Do with Them? , 2015, HEALTHINF.

[5]  M. Egger,et al.  Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study , 2017, Systematic Reviews.

[6]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study , 2016, PLoS medicine.

[7]  Wil M. P. van der Aalst,et al.  Process Mining in Healthcare: Data Challenges When Answering Frequently Posed Questions , 2012, ProHealth/KR4HC.

[8]  James Thomas,et al.  Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews , 2016, Systematic Reviews.

[9]  M. Sculpher,et al.  Decision Modelling for Health Economic Evaluation , 2006 .

[10]  John P A Ioannidis,et al.  The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. , 2016, The Milbank quarterly.

[11]  A. J. M. M. Weijters,et al.  Flexible Heuristics Miner (FHM) , 2011, 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[12]  Byron C. Wallace,et al.  Modernizing the systematic review process to inform comparative effectiveness: tools and methods. , 2013, Journal of comparative effectiveness research.

[13]  David Moher,et al.  Evidence summaries: the evolution of a rapid review approach , 2012, Systematic Reviews.

[14]  H. Bastian,et al.  Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? , 2010, PLoS medicine.

[15]  J. R. Murphy,et al.  Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation. , 1996, Journal of clinical epidemiology.

[16]  P. Glasziou,et al.  Systematic review automation technologies , 2014, Systematic Reviews.

[17]  Hajo A. Reijers,et al.  Discovering Social Networks from Event Logs , 2005, Computer Supported Cooperative Work (CSCW).

[18]  Carla E. Brodley,et al.  Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining , 2012, Genetics in Medicine.

[19]  Wil M. P. van der Aalst,et al.  Process Mining - Discovery, Conformance and Enhancement of Business Processes , 2011 .

[20]  I. Olkin,et al.  Estimating time to conduct a meta-analysis from number of citations retrieved. , 1999, JAMA.

[21]  David Moher,et al.  All in the Family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more , 2015, Systematic Reviews.

[22]  Guy Tsafnat,et al.  Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR) , 2018, Systematic Reviews.

[23]  Andrew W. Brown,et al.  Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry , 2017, BMJ Open.

[24]  D. Moher,et al.  A scoping review of rapid review methods , 2015, BMC Medicine.

[25]  S. Iliffe,et al.  The impact of Cochrane Reviews: a mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research. , 2015, Health technology assessment.

[26]  Siddhartha R. Jonnalagadda,et al.  Automating data extraction in systematic reviews: a systematic review , 2015, Systematic Reviews.

[27]  S. Ananiadou,et al.  Using text mining for study identification in systematic reviews: a systematic review of current approaches , 2015, Systematic Reviews.

[28]  Naomie Salim,et al.  Systematic literature review (SLR) automation: a systematic literature review , 2014 .

[29]  Phil Edwards,et al.  Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records , 2002, Statistics in medicine.

[30]  Boudewijn F. van Dongen,et al.  The ProM Framework: A New Era in Process Mining Tool Support , 2005, ICATPN.

[31]  Sharon E. Straus,et al.  Interventions to improve the use of systematic reviews in decision-making by health system managers, policy makers and clinicians. , 2012, The Cochrane database of systematic reviews.

[32]  W. Ungar,et al.  An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines , 2006, Public Health Nutrition.

[33]  Lisa Hartling,et al.  Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. , 2010, Journal of clinical epidemiology.

[34]  Wil M. P. van der Aalst,et al.  Process mining: a research agenda , 2004, Comput. Ind..

[35]  DATA PRESENTATION Box-plots for microbiologists? , 1993, The Lancet.

[36]  Byron C. Wallace,et al.  Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial , 2016, Systematic Reviews.

[37]  Sharon E. Straus,et al.  Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review , 2015, Implementation Science.

[38]  Jorge Munoz-Gama,et al.  Process mining in healthcare: A literature review , 2016, J. Biomed. Informatics.

[39]  Enrico Coiera,et al.  The automation of systematic reviews , 2013, BMJ.