Distilling Information from a Flood: A Possibility for the Use of Meta-Analysis and Systematic Review in Machine Learning Research

The current flood of information in all areas of machine learning research, from computer vision to reinforcement learning, has made it difficult to make aggregate scientific inferences. It can be challenging to distill a myriad of similar papers into a set of useful principles, to determine which new methodologies to use for a particular application, and to be confident that one has compared against all relevant related work when developing new ideas. However, such a rapidly growing body of research literature is a problem that other fields have already faced - in particular, medicine and epidemiology. In those fields, systematic reviews and meta-analyses have been used exactly for dealing with these issues and it is not uncommon for entire journals to be dedicated to such analyses. Here, we suggest the field of machine learning might similarly benefit from meta-analysis and systematic review, and we encourage further discussion and development along this direction.

[1]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[2]  Madelon van Wely,et al.  The good, the bad and the ugly: meta-analyses , 2014 .

[3]  Jean-Marie Chauvet,et al.  The 30-Year Cycle In The AI Debate , 2018, ArXiv.

[4]  D. Sculley,et al.  Winner's Curse? On Pace, Progress, and Empirical Rigor , 2018, ICLR.

[5]  A B Haidich,et al.  Meta-analysis in medical research. , 2010, Hippokratia.

[6]  Angela R. Laird,et al.  Ten simple rules for neuroimaging meta-analysis , 2018, Neuroscience & Biobehavioral Reviews.

[7]  Paulo J. G. Lisboa,et al.  The Use of Artificial Neural Networks in Decision Support in Cancer: a Systematic Review , 2005 .

[8]  S. Gopalakrishnan,et al.  Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare , 2013, Journal of family medicine and primary care.

[9]  L. Simpson.  Report on Certain Enteric Fever Inoculation Statistics , 1904, British medical journal.

[10]  H J Eysenck,et al.  Systematic Reviews: Meta-analysis and its problems , 1994, BMJ.

[11]  John P A Ioannidis,et al.  The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. , 2016, The Milbank quarterly.

[12]  Jascha Sohl-Dickstein,et al.  Measuring the Effects of Data Parallelism on Neural Network Training , 2018, J. Mach. Learn. Res..

[13]  Jan Peters,et al.  Reinforcement learning in robotics: A survey , 2013, Int. J. Robotics Res..

[14]  K. O'rourke,et al.  An historical perspective on meta-analysis: Dealing quantitatively with varying study results , 2007, Journal of the Royal Society of Medicine.

[15]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[16]  Manojit Chattopadhyay,et al.  A Comprehensive Review and Meta-Analysis on Applications of Machine Learning Techniques in Intrusion Detection , 2018, Australas. J. Inf. Syst..

[17]  Alfonso Rojas Espinosa,et al.  A Meta-analysis on Classification Model Performance in Real-World Datasets: An Exploratory View , 2017, Appl. Artif. Intell..

[18]  John Beatty,et al.  The Empire of Chance: How Probability Changed Science and Everyday Life , 1989 .

[19]  Jessica Gurevitch,et al.  Meta-analysis and the science of research synthesis , 2018, Nature.

[20]  Yuxi Li,et al.  Deep Reinforcement Learning: An Overview , 2017, ArXiv.

[21]  Chris Dyer,et al.  On the State of the Art of Evaluation in Neural Language Models , 2017, ICLR.