Addressing bias in big data and AI for health care: A call for open science

Summary Artificial intelligence (AI) has an astonishing potential in assisting clinical decision making and revolutionizing the field of health care. A major open challenge that AI will need to address before its integration in the clinical routine is that of algorithmic bias. Most AI algorithms need big datasets to learn from, but several groups of the human population have a long history of being absent or misrepresented in existing biomedical datasets. If the training data is misrepresentative of the population variability, AI is prone to reinforcing bias, which can lead to fatal outcomes, misdiagnoses, and lack of generalization. Here, we describe the challenges in rendering AI algorithms fairer, and we propose concrete steps for addressing bias using tools from the field of open science.

[1]  M. Tevfik Dorak,et al.  Gender Differences in Cancer Susceptibility: An Inadequately Addressed Issue , 2012, Front. Gene..

[2]  Nicolette de Keizer,et al.  Guideline for good evaluation practice in health informatics (GEP-HI) , 2011, Int. J. Medical Informatics.

[3]  Abderrahim Beni Hssane,et al.  Big healthcare data: preserving security and privacy , 2018, Journal of Big Data.

[4]  Andrea Cornwall,et al.  What is participatory research? , 1995, Social science & medicine.

[5]  Leo Anthony Celi,et al.  The “inconvenient truth” about AI in healthcare , 2019, npj Digital Medicine.

[6]  Catherine E. Chronaki,et al.  Standards in Healthcare Data , 2018, Fundamentals of Clinical Data Science.

[7]  Wilson Liao,et al.  Machine Learning in Dermatology: Current Applications, Opportunities, and Limitations , 2020, Dermatology and Therapy.

[8]  Mark Briers,et al.  The epidemiological impact of the NHS COVID-19 app , 2021, Nature.

[9]  Dana Lewis,et al.  Real-World Use of Open Source Artificial Pancreas Systems , 2016, Journal of diabetes science and technology.

[10]  N. Tatonetti,et al.  Using Machine Learning to Identify Adverse Drug Effects Posing Increased Risk to Women , 2020, Patterns.

[11]  S. Korkmaz,et al.  Sex differences in drugs: the development of a comprehensive knowledge base to improve gender awareness prescribing , 2017, Biology of Sex Differences.

[12]  Francis S Collins,et al.  Policy: NIH to balance sex in cell and animal studies , 2014, Nature.

[13]  Athina Tzovara,et al.  The Personal Data Is Political , 2018, Philosophical Studies Series.

[14]  Martha J. Farah,et al.  Socioeconomic status and the brain: mechanistic insights from human and animal research , 2010, Nature Reviews Neuroscience.

[15]  S. Fullerton,et al.  Genomics is failing on diversity , 2016, Nature.

[16]  Yunfeng Zhang,et al.  AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias , 2019, IBM Journal of Research and Development.

[17]  J. Manson,et al.  Association of Race and Ethnicity With Late-Life Depression Severity, Symptom Burden, and Care. , 2020, JAMA network open.

[18]  Y. Appelman,et al.  Gender differences in coronary heart disease , 2010, Netherlands heart journal : monthly journal of the Netherlands Society of Cardiology and the Netherlands Heart Foundation.

[19]  Clémence Réda,et al.  Machine learning applications in drug development , 2019, Computational and structural biotechnology journal.

[20]  A. Mazumder,et al.  Does “AI” stand for augmenting inequality in the era of covid-19 healthcare? , 2021, BMJ.

[21]  Ravi B. Parikh,et al.  Addressing Bias in Artificial Intelligence in Health Care. , 2019, JAMA.

[22]  Brian W. Powers,et al.  Dissecting racial bias in an algorithm used to manage the health of populations , 2019, Science.

[23]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[24]  Hamed Jalaly Bidgoly,et al.  A survey on methods and challenges in EEG based authentication , 2020, Comput. Secur..

[25]  Wojciech Samek,et al.  Methods for interpreting and understanding deep neural networks , 2017, Digit. Signal Process..

[26]  John W. Severinghaus,et al.  Effects of Skin Pigmentation on Pulse Oximeter Accuracy at Low Saturation , 2005, Anesthesiology.

[27]  J. Mckinlay,et al.  Disparities in physicians' interpretations of heart disease symptoms by patient gender: results of a video vignette factorial experiment. , 2009, Journal of women's health.

[28]  Adolph Reed,et al.  Racial Health Disparities and Covid-19 - Caution and Context. , 2020, The New England journal of medicine.

[29]  Tim Holland-Letz,et al.  Deep neural networks are superior to dermatologists in melanoma image classification. , 2019, European journal of cancer.

[30]  Michał Grochowski,et al.  Data augmentation for improving deep learning in image classification problem , 2018, 2018 International Interdisciplinary PhD Workshop (IIPhDW).

[31]  T. Lancet Meeting the unique health-care needs of LGBTQ people , 2016, The Lancet.

[32]  Miroslav Dudík,et al.  Fair Regression: Quantitative Definitions and Reduction-based Algorithms , 2019, ICML.

[33]  T. Penzel,et al.  Computer based sleep recording and analysis. , 2000, Sleep medicine reviews.

[34]  Laurel Smith-Doerr,et al.  How Diversity Matters in the US Science and Engineering Workforce: A Critical Review Considering Integration in Teams, Fields, and Organizational Contexts , 2017 .

[35]  A. Parwani Next generation diagnostic pathology: use of digital pathology and artificial intelligence tools to augment a pathological diagnosis , 2019, Diagnostic Pathology.

[36]  Karen L. Calderone The influence of gender on the frequency of pain and sedative medication administered to postoperative patients , 1990 .

[37]  N. Powe,et al.  Diversity in Clinical and Biomedical Research: A Promise Yet to Be Fulfilled , 2015, bioRxiv.

[38]  R. Cuocolo,et al.  Current applications of big data and machine learning in cardiology , 2019, Journal of geriatric cardiology : JGC.

[39]  Chris Grasso,et al.  Planning and implementing sexual orientation and gender identity data collection in electronic health records , 2018, J. Am. Medical Informatics Assoc..

[40]  T. Davenport,et al.  The potential for artificial intelligence in healthcare , 2019, Future Healthcare Journal.

[41]  E. V. van Beek,et al.  A novel machine learning-derived radiotranscriptomic signature of perivascular fat improves cardiac risk prediction using coronary CT angiography , 2019, European heart journal.

[42]  P. Anderer,et al.  Interrater reliability for sleep scoring according to the Rechtschaffen & Kales and the new AASM standard , 2009, Journal of sleep research.

[43]  Philippe N. Tobler,et al.  Cognitive biases associated with medical decisions: a systematic review , 2016, BMC Medical Informatics and Decision Making.

[44]  Regina M. Benjamin,et al.  Call to Action: Structural Racism as a Fundamental Driver of Health Disparities: A Presidential Advisory From the American Heart Association. , 2020, Circulation.

[45]  M. Kibbe,et al.  Sex bias exists in basic science and translational surgical research. , 2014, Surgery.

[46]  R. Mehrotra,et al.  Skin Cancer Concerns in People of Color: Risk Factors and Prevention , 2016, Asian Pacific journal of cancer prevention : APJCP.

[47]  I. Gotlib,et al.  Time-varying effects of income on hippocampal volume trajectories in adolescent girls , 2017, Developmental Cognitive Neuroscience.

[48]  N. Esnaola,et al.  Racial differences and disparities in cancer care and outcomes: where's the rub? , 2012, Surgical oncology clinics of North America.

[49]  P. Vinny,et al.  The Neurologist and Artificial Intelligence: Titans at Crossroads , 2019, Annals of Indian Academy of Neurology.

[50]  Blake Lemoine,et al.  Mitigating Unwanted Biases with Adversarial Learning , 2018, AIES.

[51]  Alessandro Puiatti,et al.  Automated sleep scoring: A review of the latest approaches. , 2019, Sleep medicine reviews.

[52]  J. Henrich,et al.  The weirdest people in the world? , 2010, Behavioral and Brain Sciences.

[53]  Oded Nov,et al.  Open Humans: A platform for participant-centered research and personal data exploration , 2018, bioRxiv.

[54]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[55]  P. Visscher,et al.  Tissue-specific sex-differences in human gene expression. , 2019, Human molecular genetics.

[56]  Ribana Roscher,et al.  Explainable Machine Learning for Scientific Insights and Discoveries , 2019, IEEE Access.

[57]  Erik T. Mitchell,et al.  Current Work in Diversity, Inclusion and Accessibility by Metadata Communities: A Working Report from the ALA/ALCTS Metadata Standards Committee , 2018 .

[58]  Eric J Topol,et al.  Welcoming new guidelines for AI clinical research , 2020, Nature Medicine.

[59]  Hermann Baumgartl,et al.  Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms , 2020, PloS one.

[60]  Mustafa Suleyman,et al.  Key challenges for delivering clinical impact with artificial intelligence , 2019, BMC Medicine.

[61]  E. C. Neto,et al.  Using permutations to detect, quantify and correct for confounding in machine learning predictions , 2018 .

[62]  D. Bowen,et al.  Sexual orientation and health: comparisons in the women's health initiative sample. , 2000, Archives of family medicine.

[63]  Michael Hobensack,et al.  Racial Bias in Pulse Oximetry , 2021 .

[64]  Micah J. Sheller,et al.  The future of digital health with federated learning. , 2020, NPJ digital medicine.

[65]  W Nicholson Price,et al.  Big data and black-box medical algorithms , 2018, Science Translational Medicine.

[66]  Farah Magrabi,et al.  Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications , 2019, Yearbook of Medical Informatics.

[67]  L. Kamulegeya,et al.  Using artificial intelligence on dermatology conditions in Uganda: a case for diversity in training data sets for machine learning , 2019, bioRxiv.

[68]  P Huston,et al.  Reaping the benefits of Open Data in public health. , 2019, Canada communicable disease report = Releve des maladies transmissibles au Canada.

[69]  Michael Wainberg,et al.  Deep learning in biomedicine , 2018, Nature Biotechnology.

[70]  R. L. Cooper,et al.  Training to reduce LGBTQ-related bias among medical, nursing, and dental students and providers: a systematic review , 2019, BMC Medical Education.

[71]  M. Sjoding,et al.  Racial Bias in Pulse Oximetry Measurement. , 2020, The New England journal of medicine.