Predictive analytics in health care: how can we know it works?

Abstract There is increasing awareness that the methodology and findings of research should be transparent. This includes studies using artificial intelligence to develop predictive algorithms that make individualized diagnostic or prognostic risk predictions. We argue that it is paramount to make the algorithm behind any prediction publicly available. This allows independent external validation, assessment of performance heterogeneity across settings and over time, and algorithm refinement or updating. Online calculators and apps may aid uptake if accompanied with sufficient information. For algorithms based on “black box” machine learning methods, software for algorithm implementation is a must. Hiding algorithms for commercial exploitation is unethical, because there is no possibility to assess whether algorithms work as advertised or to monitor when and how algorithms are updated. Journals and funders should demand maximal transparency for publications on predictive algorithms, and clinical guidelines should only recommend publicly available algorithms.

[1]  I. Kohane,et al.  Big Data and Machine Learning in Health Care. , 2018, JAMA.

[2]  Seong Ho Park,et al.  Regulatory Approval versus Clinical Validation of Artificial Intelligence Diagnostic Tools. , 2018, Radiology.

[3]  Conor Liston,et al.  New machine-learning technologies for computer-aided diagnosis , 2018, Nature Medicine.

[4]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[5]  Jennifer G. Robinson,et al.  2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk: A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines , 2014, Circulation.

[6]  D. Mark,et al.  Clinical prediction models: are we building better mousetraps? , 2003, Journal of the American College of Cardiology.

[7]  Lucila Ohno-Machado,et al.  Data science and artificial intelligence to improve clinical practice and research , 2018, J. Am. Medical Informatics Assoc..

[8]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[9]  Yvonne Vergouwe,et al.  A calibration hierarchy for risk models was defined: from utopia to empirical data. , 2016, Journal of clinical epidemiology.

[10]  Noémie Elhadad,et al.  When to re-order laboratory tests? Learning laboratory test shelf-life , 2018, J. Biomed. Informatics.

[11]  John P. A. Ioannidis,et al.  Reproducible Research Practices and Transparency across the Biomedical Literature , 2016, PLoS biology.

[12]  Rae Woong Park,et al.  Characterizing treatment pathways at scale using the OHDSI network , 2016, Proceedings of the National Academy of Sciences.

[13]  Martha Sajatovic,et al.  Clinical Prediction Models , 2013 .

[14]  Jie Ma,et al.  A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. , 2019, Journal of clinical epidemiology.

[15]  Russ B. Altman,et al.  Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets , 2017, Int. J. Medical Informatics.

[16]  Jie Xu,et al.  The practical implementation of artificial intelligence technologies in medicine , 2019, Nature Medicine.

[17]  David Moher,et al.  Reducing waste from incomplete or unusable reports of biomedical research , 2014, The Lancet.

[18]  D Timmerman,et al.  Strategies to diagnose ovarian cancer: new evidence from phase 3 of the multicentre international IOTA study , 2014, British Journal of Cancer.

[19]  L. Nevin.  Advancing the beneficial use of machine learning in health care and medicine: Toward a community understanding , 2018, PLoS medicine.

[20]  E. Vayena,et al.  Machine learning in medicine: Addressing ethical challenges , 2018, PLoS medicine.

[21]  Gary S Collins,et al.  Fracture Risk Assessment: State of the Art, Methodologically Unsound, or Poorly Reported? , 2012, Current Osteoporosis Reports.

[22]  George Hripcsak,et al.  Next-generation phenotyping of electronic health records , 2012, J. Am. Medical Informatics Assoc..

[23]  Patrick B. Ryan,et al.  Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data , 2018, J. Am. Medical Informatics Assoc..

[24]  D. Avard,et al.  Legal Liability and the Uncertain Nature of Risk Prediction: The Case of Breast Cancer Risk Prediction Models , 2012, Public Health Genomics.

[25]  Gary S Collins,et al.  Prognostic models in obstetrics: available, but far from applicable. , 2016, American journal of obstetrics and gynecology.

[26]  N. Shah,et al.  Implementing Machine Learning in Health Care - Addressing Ethical Challenges. , 2018, The New England journal of medicine.

[27]  Guanhua Chen,et al.  Calibration drift in regression and machine learning models for acute kidney injury , 2017, J. Am. Medical Informatics Assoc..

[28]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[29]  Thomas Wiegand,et al.  WHO and ITU establish benchmarking process for artificial intelligence in health , 2019, The Lancet.

[30]  Gary S Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration , 2015, Annals of Internal Medicine.

[31]  Jennifer G. Robinson,et al.  2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk: A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines , 2014, Circulation.

[32]  Michael J Pencina,et al.  Risk Prediction With Electronic Health Records: The Importance of Model Validation and Clinical Context. , 2016, JAMA cardiology.

[33]  Ewout W Steyerberg,et al.  Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury. , 2016, Journal of clinical epidemiology.

[34]  Ewout W. Steyerberg,et al.  Big Data and Predictive Analytics: Recalibrating Expectations , 2018, JAMA.