Return on Investment in Machine Learning: Crossing the Chasm between Academia and Business

Abstract Academia remains the central place of machine learning education. While academic culture is the predominant factor influencing the way we teach machine learning to students, many practitioners question this culture, claiming the lack of alignment between academic and business environments. Drawing on professional experiences from both sides of the chasm, we describe the main points of contention, in the hope that it will help better align academic syllabi with the expectations towards future machine learning practitioners. We also provide recommendations for teaching of the applied aspects of machine learning.

[1]  Padhraic Smyth,et al.  From Data Mining to Knowledge Discovery in Databases , 1996, AI Mag..

[2]  Larry Kerschberg,et al.  A methodology and life cycle model for data mining and knowledge discovery in precision agriculture , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[3]  Stephen R. Gardner.  Building the data warehouse , 1998, CACM.

[4]  Rüdiger Wirth,et al.  CRISP-DM: Towards a Standard Process Model for Data Mining , 2000 .

[5]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[6]  Paulo C. M. Gottgtroy,et al.  Ontology Driven Knowledge Discovery Process: a proposal to integrate Ontology Engineering and KDD , 2007, PACIS.

[7]  Randall Matignon.  Data Mining Using SAS® Enterprise Miner™: Matignon/Data Mining , 2007 .

[8]  Randall Matignon,et al.  Data Mining Using SAS Enterprise Miner , 2007 .

[9]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[10]  Panos Vassiliadis.  A Survey of Extract-Transform-Load Technology , 2009, Int. J. Data Warehous. Min..

[11]  James Pustejovsky,et al.  Natural Language Annotation for Machine Learning - a Guide to Corpus-Building for Applications , 2012 .

[12]  Kiri Wagstaff,et al.  Machine Learning that Matters , 2012, ICML.

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[15]  D. Lazer,et al.  The Parable of Google Flu: Traps in Big Data Analysis , 2014, Science.

[16]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[17]  Maciej Drozdowski,et al.  Mind the gap: a heuristic study of subway tours , 2014, J. Heuristics.

[18]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[19]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[20]  D. Sculley,et al.  Hidden Technical Debt in Machine Learning Systems , 2015, NIPS.

[21]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[22]  D. Sculley,et al.  What’s your ML test score? A rubric for ML production systems , 2016 .

[23]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[24]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[25]  Megan Garcia,et al.  Racist in the Machine: The Disturbing Implications of Algorithmic Bias , 2016 .

[26]  Christopher Ré,et al.  Snorkel: Rapid Training Data Creation with Weak Supervision , 2017, Proc. VLDB Endow..

[27]  Jason S. Kessler,et al.  Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ , 2017, ACL.

[28]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[29]  Andrew Zisserman,et al.  Multi-task Self-Supervised Visual Learning , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Rachael Tatman,et al.  Gender and Dialect Bias in YouTube’s Automatic Captions , 2017, EthNLP@EACL.

[31]  Jian Yang,et al.  Learning with Inadequate and Incorrect Supervision , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[32]  Jaap Kamps,et al.  Learning to Learn from Weak Supervision by Full Supervision , 2017, ArXiv.

[33]  Virginia Dignum,et al.  Responsible Artificial Intelligence: Designing Ai for Human Values , 2017 .

[34]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[35]  Ayanna M. Howard,et al.  The Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity , 2017, Science and Engineering Ethics.

[36]  Miryung Kim,et al.  Data Scientists in Software Teams: State of the Art and Challenges , 2018, IEEE Transactions on Software Engineering.

[37]  Hwee Tou Ng,et al.  Upping the Ante: Towards a Better Benchmark for Chinese-to-English Machine Translation , 2018, LREC.

[38]  Christopher Potts,et al.  Mittens: an Extension of GloVe for Learning Domain-Specialized Representations , 2018, NAACL.

[39]  Denis Turdakov,et al.  Active Learning and Crowdsourcing: A Survey of Optimization Methods for Data Labeling , 2018, Programming and Computer Software.

[40]  Pablo Estevez,et al.  150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com , 2019, KDD.

[41]  Andreas Holzinger,et al.  Biomedical image augmentation using Augmentor , 2019, Bioinform..

[42]  Thomas Wolf,et al.  Transfer Learning in Natural Language Processing , 2019, NAACL.

[43]  Alexander M. Rush,et al.  Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models , 2018, IEEE Transactions on Visualization and Computer Graphics.

[44]  Holger H. Hoos,et al.  A survey on semi-supervised learning , 2019, Machine Learning.

[45]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[46]  Hao Yang,et al.  DQNViz: A Visual Analytics Approach to Understand Deep Q-Networks , 2019, IEEE Transactions on Visualization and Computer Graphics.

[47]  Sendhil Mullainathan,et al.  Dissecting Racial Bias in an Algorithm that Guides Health Decisions for 70 Million People , 2019, FAT.

[48]  Ion Stoica,et al.  Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules , 2019, ICML.

[49]  Jean-Pierre Lorré,et al.  Weak Supervision for Learning Discourse Structure , 2019, EMNLP.

[50]  Alexandre Lacoste,et al.  Quantifying the Carbon Emissions of Machine Learning , 2019, ArXiv.

[51]  Michael Gofman,et al.  Artificial Intelligence, Human Capital, and Innovation , 2020, SSRN Electronic Journal.

[52]  Dawn Song,et al.  Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty , 2019, NeurIPS.

[53]  Jeremy Cherfas.  TLDRLegal - Software Licenses Explained in Plain English , 2019 .

[54]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[55]  Ivica Crnkovic,et al.  Engineering AI Systems: A Research Agenda , 2020, Advances in Systems Analysis, Software Engineering, and High Performance Computing.

[56]  Shandong Wu,et al.  Inaccurate Labels in Weakly-Supervised Deep Learning: Automatic Identification and Correction and Their Impact on Classification Performance , 2020, IEEE Journal of Biomedical and Health Informatics.

[57]  Sanjai Rayadurgam,et al.  Manifold for Machine Learning Assurance , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER).

[58]  Alexandr A. Kalinin,et al.  Albumentations: fast and flexible image augmentations , 2018, Inf..

[59]  Sameer Singh,et al.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList , 2020, ACL.

[60]  Bryan Hosack.  You look like a thing and i love you: how artificial intelligence works and why it’s making the world a weirder place , 2020 .

[61]  Anna Zaitsev,et al.  Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization , 2019, First Monday.

[62]  Improving Undergraduate STEM Education: Hispanic‐Serving Institutions , 2020, The National Teaching & Learning Forum.

[63]  David J. Hand,et al.  Validating and Verifying AI Systems , 2020, Patterns.

[64]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[65]  Hasan Ferit Eniser,et al.  Importance-Driven Deep Learning System Testing , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[66]  Fei Fang,et al.  Artificial Intelligence for Social Good: A Survey , 2020, ArXiv.

[67]  Jie M. Zhang,et al.  Automatic Testing and Improvement of Machine Translation , 2019, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[68]  Li Wang,et al.  ARTDL: Adaptive Random Testing for Deep Learning Systems , 2020, IEEE Access.

[69]  Samet Demir,et al.  DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning , 2019, AISafety@IJCAI.

[70]  Steven Euijong Whang,et al.  A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective , 2018, IEEE Transactions on Knowledge and Data Engineering.

[71]  Gail C. Murphy,et al.  How does Machine Learning Change Software Development Practices? , 2021, IEEE Transactions on Software Engineering.

[72]  Mark Harman,et al.  Machine Learning Testing: Survey, Landscapes and Horizons , 2019, IEEE Transactions on Software Engineering.