How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey

There is increasing interest in machine learning (ML) techniques and their applications in recent years. Although there has been intensive support by frameworks and libraries for the implementation of ML-based systems, investigation into engineering disciplines and methods is still at the early phase. The most pressing issue in this field is identifying the essential challenges for the software engineering research community as engineering of ML-based systems requires novel approaches due to the essentially different nature of ML-based systems. In this paper, we analyze the results of a questionnaire administered to 278 people who have worked on ML-based systems in practice, clarify the essential difficulties and their causes as perceived by practitioners, and suggest potential research directions.

[1]  Matthew Mirman,et al.  Differentiable Abstract Interpretation for Provably Robust Neural Networks , 2018, ICML.

[2]  Sanjit A. Seshia,et al.  Compositional Falsification of Cyber-Physical Systems with Machine Learning Components , 2017, NFM.

[3]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[4]  Wei Cai,et al.  A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View , 2018, IEEE Access.

[5]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[6]  Sergio Segura,et al.  A Survey on Metamorphic Testing , 2016, IEEE Transactions on Software Engineering.

[7]  W. B. Roberts,et al.  Machine Learning: The High Interest Credit Card of Technical Debt , 2014 .

[8]  D. Sculley,et al.  What’s your ML test score? A rubric for ML production systems , 2016 .

[9]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[10]  David Gunning,et al.  DARPA's explainable artificial intelligence (XAI) program , 2019, IUI.

[11]  R. P. Jagadeesh Chandra Bose,et al.  Identifying implementation bugs in machine learning based image classifiers using metamorphic testing , 2018, ISSTA.

[12]  Junfeng Yang,et al.  DeepXplore , 2019, Commun. ACM.

[13]  Neoklis Polyzotis,et al.  Data Management Challenges in Production Machine Learning , 2017, SIGMOD Conference.

[14]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[15]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.