Automated machine learning: Review of the state-of-the-art and opportunities for healthcare

OBJECTIVE This work aims to provide a review of the existing literature in the field of automated machine learning (AutoML) to help healthcare professionals better utilize machine learning models "off-the-shelf" with limited data science expertise. We also identify the potential opportunities and barriers to using AutoML in healthcare, as well as existing applications of AutoML in healthcare. METHODS Published papers, accompanied with code, describing work in the field of AutoML from both a computer science perspective or a biomedical informatics perspective were reviewed. We also provide a short summary of a series of AutoML challenges hosted by ChaLearn. RESULTS A review of 101 papers in the field of AutoML revealed that these automated techniques can match or improve upon expert human performance in certain machine learning tasks, often in a shorter amount of time. The main limitation of AutoML at this point is the ability to get these systems to work efficiently on a large scale, i.e. beyond small- and medium-size retrospective datasets. DISCUSSION The utilization of machine learning techniques has the demonstrated potential to improve health outcomes, cut healthcare costs, and advance clinical research. However, most hospitals are not currently deploying machine learning solutions. One reason for this is that health care professionals often lack the machine learning expertise that is necessary to build a successful model, deploy it in production, and integrate it with the clinical workflow. In order to make machine learning techniques easier to apply and to reduce the demand for human experts, automated machine learning (AutoML) has emerged as a growing field that seeks to automatically select, compose, and parametrize machine learning models, so as to achieve optimal performance on a given task and/or dataset. CONCLUSION While there have already been some use cases of AutoML in the healthcare field, more work needs to be done in order for there to be widespread adoption of AutoML in healthcare.

[1]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[2]  Deepak S. Turaga,et al.  Learning Feature Engineering for Classification , 2017, IJCAI.

[3]  Eyke Hüllermeier,et al.  (WIP) Towards the Automated Composition of Machine Learning Services , 2018, 2018 IEEE International Conference on Services Computing (SCC).

[4]  J. H. Rudd,et al.  Cardiovascular disease risk prediction using automated machine learning: A prospective study of 423,604 UK Biobank participants , 2019, PloS one.

[5]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[6]  Ramesh Raskar,et al.  Accelerating Neural Architecture Search using Performance Prediction , 2017, ICLR.

[7]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[8]  Frank Hutter,et al.  Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves , 2015, IJCAI.

[9]  Yoshua Bengio,et al.  On Using Very Large Target Vocabulary for Neural Machine Translation , 2014, ACL.

[10]  M. Weinstock,et al.  Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and reproducibility study , 2017, British Medical Journal.

[11]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[12]  Prakash M. Nadkarni,et al.  Guidelines for the effective use of entity-attribute-value modeling for biomedical databases , 2007, Int. J. Medical Informatics.

[13]  Mihaela van der Schaar,et al.  Prognostication and Risk Factors for Cystic Fibrosis via Automated Machine Learning , 2018, Scientific Reports.

[14]  Magnus Rattray,et al.  Making sense of big data in health research: Towards an EU action plan , 2016, Genome Medicine.

[15]  Dana S. Nau,et al.  SHOP2: An HTN Planning System , 2003, J. Artif. Intell. Res..

[16]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[17]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[18]  Jeffrey Dean,et al.  Machine Learning in Medicine , 2019, The New England journal of medicine.

[19]  Arthur W. Toga,et al.  Big biomedical data as the key resource for discovery science , 2015, J. Am. Medical Informatics Assoc..

[20]  Ameet Talwalkar,et al.  Non-stochastic Best Arm Identification and Hyperparameter Optimization , 2015, AISTATS.

[21]  Xin Sun,et al.  Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence , 2019, Nature Medicine.

[22]  B. Frey,et al.  Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning , 2015, Nature Biotechnology.

[23]  Xi Li,et al.  GNAS: A Greedy Neural Architecture Search Method for Multi-Attribute Learning , 2018, ACM Multimedia.

[24]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[25]  Hod Lipson,et al.  Autostacker: a compositional evolutionary learning system , 2018, GECCO.

[26]  Michèle Sebag,et al.  AutoML with Monte Carlo Tree Search , 2018, IJCAI 2018.

[27]  Kemal Kilic,et al.  An interactive machine-learning-based electronic fraud and abuse detection system in healthcare insurance , 2015, Appl. Soft Comput..

[28]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[29]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[30]  Matthew J. Hausknecht,et al.  Beyond short snippets: Deep networks for video classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[32]  Tim Kraska,et al.  Automating model search for large scale machine learning , 2015, SoCC.

[33]  S. Perera,et al.  Using Machine Learning to Examine Medication Adherence Thresholds and Risk of Hospitalization , 2015, Medical care.

[34]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[35]  Jake Luo,et al.  Big Data Application in Biomedical Research and Health Care: A Literature Review , 2016, Biomedical informatics insights.

[36]  Gang Luo,et al.  Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection , 2017, Health Inf. Sci. Syst..

[37]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[38]  Tim Kraska,et al.  MLbase: A Distributed Machine-learning System , 2013, CIDR.

[39]  Lars Schmidt-Thieme,et al.  Automatic Frankensteining: Creating Complex Ensembles Autonomously , 2017, SDM.

[40]  Quoc V. Le,et al.  Understanding and Simplifying One-Shot Architecture Search , 2018, ICML.

[41]  Kuan-Ta Chen,et al.  Automation of the kidney function prediction and classification through ultrasound-based kidney imaging using deep learning , 2019, npj Digital Medicine.

[42]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[43]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[44]  Tim Oates,et al.  Efficient progressive sampling , 1999, KDD '99.

[45]  Risto Miikkulainen,et al.  Designing neural networks through neuroevolution , 2019, Nat. Mach. Intell..

[46]  Gang Luo,et al.  Automatically explaining machine learning prediction results: a demonstration on type 2 diabetes risk prediction , 2016, Health Inf. Sci. Syst..

[47]  J. Rumsfeld,et al.  Big data analytics to improve cardiovascular care: promise and challenges , 2016, Nature Reviews Cardiology.

[48]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[49]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[50]  Marc Parizeau,et al.  DEAP: evolutionary algorithms made easy , 2012, J. Mach. Learn. Res..

[51]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[52]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[53]  Kalyan Veeramachaneni,et al.  Deep feature synthesis: Towards automating data science endeavors , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[54]  Aaron Klein,et al.  Towards Automatically-Tuned Neural Networks , 2016, AutoML@ICML.

[55]  Deepak S. Turaga,et al.  Feature Engineering for Predictive Modeling using Reinforcement Learning , 2017, AAAI.

[56]  Prasanna Balaprakash,et al.  DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks , 2018, 2018 IEEE 25th International Conference on High Performance Computing (HiPC).

[57]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[58]  Arun Ross,et al.  ATM: A distributed, collaborative, scalable system for automated machine learning , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[59]  Gang Luo,et al.  A review of automatic selection methods for machine learning algorithms and hyper-parameter values , 2016, Network Modeling Analysis in Health Informatics and Bioinformatics.

[60]  Gang Luo,et al.  MLBCD: a machine learning tool for big clinical data , 2015, Health Inf. Sci. Syst..

[61]  Deepak S. Turaga,et al.  Cognito: Automated Feature Engineering for Supervised Learning , 2016, 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW).

[62]  Patricia Kipnis,et al.  Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals. , 2016, Journal of hospital medicine.

[63]  T. Murdoch,et al.  The inevitable application of big data to health care. , 2013, JAMA.

[64]  Eyke Hüllermeier,et al.  ML-Plan: Automated machine learning via hierarchical planning , 2018, Machine Learning.

[65]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[66]  George Hripcsak,et al.  Caveats for the use of operational electronic health record data in comparative effectiveness research. , 2013, Medical care.

[67]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[68]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[69]  Vikram Pudi,et al.  AutoLearn — Automated Feature Generation and Selection , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[70]  Sergio Escalera,et al.  Analysis of the AutoML Challenge Series 2015-2018 , 2019, Automated Machine Learning.

[71]  Prabhat,et al.  Scalable Bayesian Optimization Using Deep Neural Networks , 2015, ICML.

[72]  Paolo Traverso,et al.  Automated planning - theory and practice , 2004 .

[73]  Tara N. Sainath,et al.  Improving deep neural networks for LVCSR using rectified linear units and dropout , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[74]  Lars Schmidt-Thieme,et al.  Scalable Gaussian process-based transfer surrogates for hyperparameter optimization , 2017, Machine Learning.

[75]  Gang Luo,et al.  PredicT-ML: a tool for automating machine learning model building with big clinical data , 2016, Health Information Science and Systems.

[76]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[77]  Yong Yu,et al.  Efficient Architecture Search by Network Transformation , 2017, AAAI.

[78]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[79]  Ehsaneddin Asgari,et al.  Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics , 2015, PloS one.

[80]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[81]  G. Corrado,et al.  End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography , 2019, Nature Medicine.

[82]  Daniel Svozil,et al.  Introduction to multi-layer feed-forward neural networks , 1997 .

[83]  N. Luhmann.  Trust and Power , 1979 .

[84]  Dawn Xiaodong Song,et al.  ExploreKit: Automatic Feature Generation and Selection , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[85]  Kevin Leyton-Brown,et al.  An Efficient Approach for Assessing Hyperparameter Importance , 2014, ICML.

[86]  Hugo Jair Escalante,et al.  Particle Swarm Model Selection , 2009, J. Mach. Learn. Res..

[87]  I. Kohane,et al.  Big Data and Machine Learning in Health Care. , 2018, JAMA.

[88]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.

[89]  Kotaro Hirasawa,et al.  Multi-branch structure of layered neural networks , 2002, Proceedings of the 41st SICE Annual Conference. SICE 2002..

[90]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[91]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[92]  Frank Hutter,et al.  Neural Architecture Search: A Survey , 2018, J. Mach. Learn. Res..

[93]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  Randal S. Olson,et al.  Automating Biomedical Data Science Through Tree-Based Pipeline Optimization , 2016, EvoApplications.

[95]  Elliot Meyerson,et al.  Evolving Deep Neural Networks , 2017, Artificial Intelligence in the Age of Neural Networks and Brain Computing.

[96]  David D. Cox,et al.  Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[97]  François Laviolette,et al.  Agnostic Bayesian Learning of Ensembles , 2014, ICML.

[98]  Kalyan Veeramachaneni,et al.  FeatureHub: Towards Collaborative Data Science , 2017, 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[99]  Tie-Yan Liu,et al.  Neural Architecture Optimization , 2018, NeurIPS.

[100]  Jeffrey Dean,et al.  Scalable and accurate deep learning with electronic health records , 2018, npj Digital Medicine.

[101]  William S Weintraub,et al.  Translational Medicine in the Era of Big Data and Machine Learning. , 2018, Circulation research.

[102]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[103]  J. March.  Exploration and exploitation in organizational learning , 1991, STUDI ORGANIZZATIVI.

[104]  Chris Eliasmith,et al.  Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn , 2014, SciPy.

[105]  Billur Barshan,et al.  Detecting Falls with Wearable Sensors Using Machine Learning Techniques , 2014, Sensors.

[106]  Erin Sparnon,et al.  Screening Electronic Health Record–Related Patient Safety Reports Using Machine Learning , 2017, Journal of patient safety.

[107]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[108]  Changhu Wang,et al.  Network Morphism , 2016, ICML.

[109]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[110]  Mengjie Zhang,et al.  Genetic programming for feature construction and selection in classification on high-dimensional data , 2016, Memetic Comput..

[111]  D. Bates,et al.  Big data in health care: using analytics to identify and manage high-risk and high-cost patients. , 2014, Health affairs.

[112]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[113]  Frank Hutter,et al.  Initializing Bayesian Hyperparameter Optimization via Meta-Learning , 2015, AAAI.

[114]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[115]  Li Li,et al.  Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records , 2016, Scientific Reports.

[116]  Scott M. Lundberg,et al.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery , 2018, Nature Biomedical Engineering.

[117]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[118]  Sergio Escalera,et al.  Design of the 2015 ChaLearn AutoML challenge , 2015, IJCNN.

[119]  Wei Wu,et al.  Practical Block-Wise Neural Network Architecture Generation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[120]  Mehryar Mohri,et al.  AdaNet: Adaptive Structural Learning of Artificial Neural Networks , 2016, ICML.

[121]  Ian H. Witten,et al.  WEKA: a machine learning workbench , 1994, Proceedings of ANZIIS '94 - Australian New Zealnd Intelligent Information Systems Conference.

[122]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.