ReproducedPapers.org: Openly teaching and structuring machine learning reproducibility

We present this http URL: an open online repository for teaching and structuring machine learning reproducibility. We evaluate doing a reproduction project among students and the added value of an online reproduction repository among AI researchers. We use anonymous self-assessment surveys and obtained 144 responses. Results suggest that students who do a reproduction project place more value on scientific reproductions and become more critical thinkers. Students and AI researchers agree that our online reproduction repository is valuable.

[1]  Kyle Gorman,et al.  We Need to Talk about Standard Splits , 2019, ACL.

[2]  Konrad Hinsen,et al.  ReScience C: A Journal for Reproducible Replications in Computational Science , 2018, RRPR.

[3]  Edward Raff,et al.  A Step Toward Quantifying Independently Reproducible Machine Learning Research , 2019, NeurIPS.

[4]  Andrew McCallum,et al.  Open Scholarship and Peer Review: a Time for Experimentation , 2013 .

[5]  André Freitas,et al.  The Diagrammatic AI Language (DIAL): Version 0.1 , 2018, ArXiv.

[6]  Zachary C. Lipton,et al.  Troubling Trends in Machine Learning Scholarship , 2018, ACM Queue.

[7]  Benjamin Recht,et al.  Do ImageNet Classifiers Generalize to ImageNet? , 2019, ICML.

[8]  Chris Dyer,et al.  On the State of the Art of Evaluation in Neural Language Models , 2017, ICLR.

[9]  Senthil Mani,et al.  DLPaper2Code: Auto-generation of Code from Deep Learning Research Papers , 2017, AAAI.

[10]  Dietmar Jannach,et al.  Are we really making much progress? A worrying analysis of recent neural recommendation approaches , 2019, RecSys.

[11]  D. Sculley,et al.  Winner's Curse? On Pace, Progress, and Empirical Rigor , 2018, ICLR.

[12]  David Coeurjolly,et al.  Code replicability in computer graphics , 2020, ACM Trans. Graph..

[13]  Marco Loog,et al.  Black Magic in Deep Learning: How Human Skill Impacts Network Training , 2020, BMVC.

[14]  Jimmy J. Lin,et al.  The Neural Hype and Comparisons Against Weak Baselines , 2019, SIGIR Forum.

[15]  Nick McKeown,et al.  Learning Networking by Reproducing Research Results , 2017, CCRV.

[16]  Bertrand Kerautret,et al.  An Overview of Platforms for Reproducible Research and Augmented Publications , 2018, RRPR.

[17]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[18]  Larry Rudolph,et al.  Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO , 2020, ArXiv.

[19]  Michela Paganini,et al.  dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestration , 2020, ArXiv.

[20]  C. Drummond.  Replicability is not Reproducibility:Nor is it Good Science , 2009 .

[21]  Ser-Nam Lim,et al.  A Metric Learning Reality Check , 2020, ECCV.

[22]  Jasper Snoek,et al.  Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling , 2018, ICLR.

[23]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[24]  M. Hutson.  Artificial intelligence faces reproducibility crisis. , 2018, Science.

[25]  Pascal Monasse,et al.  IPOL: A new journal for fully reproducible research; analysis of four years development , 2015, NTMS.