The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples.

OBJECTIVES To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates. STUDY DESIGN AND SETTING We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC). RESULTS We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39). CONCLUSION When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

[1]  Rabia Bashir,et al.  Software engineering principles address current problems in the systematic review ecosystem. , 2019, Journal of clinical epidemiology.

[2]  N. Taylor,et al.  Additional Physical Therapy Services Reduce Length of Stay and Improve Health Outcomes in People With Acute and Subacute Conditions: An Updated Systematic Review and Meta-Analysis. , 2018, Archives of physical medicine and rehabilitation.

[3]  Janet E. Martin,et al.  Evidence reversal-when new evidence contradicts current claims: a systematic overview review of definitions and terms. , 2018, Journal of clinical epidemiology.

[4]  D. Moher,et al.  Identifying null meta-analyses that are ripe for updating , 2003, BMC medical research methodology.

[5]  Enrico W. Coiera,et al.  A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates , 2017, J. Biomed. Informatics.

[6]  Sally Hopewell,et al.  A multicomponent decision tool for prioritising the updating of systematic reviews , 2013, BMJ.

[7]  N. Venketasubramanian,et al.  Current Status of Intravenous Thrombolysis for Acute Ischemic Stroke in Asia , 2011, International journal of stroke : official journal of the International Stroke Society.

[8]  S. Ananiadou,et al.  Using text mining for study identification in systematic reviews: a systematic review of current approaches , 2015, Systematic Reviews.

[9]  David Ogilvie,et al.  Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews , 2014, Research synthesis methods.

[10]  Sophia Ananiadou,et al.  Reducing systematic review workload through certainty-based screening , 2014, J. Biomed. Informatics.

[11]  Aaron M. Cohen,et al.  Studying the potential impact of automated document classification on scheduling a systematic review update , 2012, BMC Medical Informatics and Decision Making.

[12]  N. Taylor,et al.  Extra physical therapy reduces patient length of stay and improves functional outcomes and quality of life in people with acute or subacute conditions: a systematic review. , 2011, Archives of physical medicine and rehabilitation.

[13]  Philip S. Yu,et al.  Aggregator: a machine learning approach to identifying MEDLINE articles that derive from the same underlying clinical trial. , 2015, Methods.

[14]  Mourad Ouzzani,et al.  Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) , 2018, Systematic Reviews.

[15]  Nicola J Cooper,et al.  Evidence‐based sample size calculations based upon updated meta‐analysis , 2007, Statistics in medicine.

[16]  Paul G Shekelle,et al.  Machine Learning Versus Standard Techniques for Updating Searches for Systematic Reviews: A Diagnostic Accuracy Study , 2017, Annals of Internal Medicine.

[17]  P. Glasziou,et al.  Systematic review automation technologies , 2014, Systematic Reviews.

[18]  Alex J Sutton,et al.  An encouraging assessment of methods to inform priorities for updating systematic reviews. , 2009, Journal of clinical epidemiology.

[19]  Tony Tse,et al.  Sharing Individual Participant Data (IPD) within the Context of the Trial Reporting System (TRS) , 2016, PLoS medicine.

[20]  A. Dunn,et al.  Time-to-update of systematic reviews relative to the availability of new evidence , 2018, Systematic Reviews.

[21]  Guy Tsafnat,et al.  Automatic Evidence Retrieval for Systematic Reviews , 2014, Journal of medical Internet research.

[22]  P. Alonso-Coello,et al.  Methodological systematic review identifies major limitations in prioritization processes for updating. , 2017, Journal of clinical epidemiology.

[23]  C. Anderson,et al.  Current status of intravenous tissue plasminogen activator dosage for acute ischaemic stroke: an updated systematic review , 2018, Stroke and Vascular Neurology.

[24]  Christine Urquhart,et al.  Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. , 2016, Journal of clinical epidemiology.

[25]  John P A Ioannidis,et al.  The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. , 2016, The Milbank quarterly.

[26]  Jian-Yun Nie,et al.  Combining classifiers for robust PICO element detection , 2010, BMC Medical Informatics Decis. Mak..

[27]  David Martínez,et al.  Automatic classification of sentences to support Evidence Based Medicine , 2011, BMC Bioinformatics.

[28]  Susanne Hempel,et al.  A Pilot Study Using Machine Learning and Domain Knowledge to Facilitate Comparative Effectiveness Review Updating , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[29]  Paul G Shekelle,et al.  Assessment of a method to detect signals for updating systematic reviews , 2014, Systematic Reviews.

[30]  Didi Surian,et al.  Trial2rev: Combining machine learning and crowd-sourcing to create a shared space for updating systematic reviews , 2019, JAMIA open.

[31]  Carla E. Brodley,et al.  Semi-automated screening of biomedical citations for systematic reviews , 2010, BMC Bioinformatics.

[32]  C. Karner,et al.  Cochrane Airways Group reviews were prioritized for updating using a pragmatic approach. , 2015, Journal of clinical epidemiology.

[33]  Julian P T Higgins,et al.  Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. , 2012, Journal of clinical epidemiology.

[34]  K. Shojania,et al.  How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis , 2007, Annals of Internal Medicine.

[35]  Jau-Min Wong,et al.  PICO element detection in medical text without metadata: Are first sentences enough? , 2013, J. Biomed. Informatics.

[36]  Ebrahim Bagheri,et al.  Improving the conduct of systematic reviews: a process mining perspective. , 2018, Journal of clinical epidemiology.