Preliminary Systematic Literature Review of Machine Learning System Development Process

Previous machine learning (ML) system development research suggests that emerging software quality attributes are a concern due to the probabilistic behavior of ML systems. Assuming that detailed development processes depend on individual developers and are not discussed in detail. To help developers to standardize their ML system development processes, we conduct a preliminary systematic literature review on ML system development processes. A search query of 2358 papers identified 7 papers as well as two other papers determined in an ad-hoc review. Our findings include emphasized phases in ML system developments, frequently described practices and tailored traditional software development practices.

[1]  Christine T. Wolf.  Professional Identity and Information Use: On Becoming a Machine Learning Developer , 2019, iConference.

[2]  Rachel K. E. Bellamy,et al.  Trials and tribulations of developers of intelligent systems: A field study , 2016, 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[3]  Foutse Khomh,et al.  Software Engineering for Machine-Learning Applications: The Road Ahead , 2018, IEEE Software.

[4]  Kim Halskov,et al.  UX Design Innovation: Challenges for Working with Machine Learning as a Design Material , 2017, CHI.

[5]  Jan Bosch,et al.  Software Engineering Challenges of Deep Learning , 2018, 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[6]  Cristina V. Lopes,et al.  Aspect-oriented programming , 1999, ECOOP Workshops.

[7]  Perry R. Cook,et al.  Human model evaluation in interactive supervised learning , 2011, CHI.

[8]  Dieter Rombach,et al.  Aligning Organizations Through Measurement: The Gqm+strategies Approach , 2014 .

[9]  Femke Ongenae,et al.  A Methodology to Involve Domain Experts and Machine Learning Techniques in the Design of Human-Centered Algorithms , 2018, HWID.

[10]  Agnieszka Billewicz.  Silly Lamp: Study of a Relationship with Engaging Machine Learning Artefacts , 2018, CHI Extended Abstracts.

[11]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[12]  D. Sculley,et al.  Hidden Technical Debt in Machine Learning Systems , 2015, NIPS.

[13]  Cristina V. Lopes,et al.  Aspect-oriented programming , 1999, ECOOP Workshops.

[14]  Toshiaki Yasue,et al.  A Survey of Software Quality for Machine Learning Applications , 2018, 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[15]  Hui Zhao,et al.  Design Concerns for Industrial Big Data System in the Smart Factory Domain: From Product Lifecycle View , 2018, 2018 23rd International Conference on Engineering of Complex Computer Systems (ICECCS).

[16]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[17]  Miryung Kim,et al.  The Emerging Role of Data Scientists on Software Development Teams , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[18]  Ivica Crnkovic,et al.  A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation , 2019, XP.

[19]  David Lo,et al.  An Empirical Study of Bugs in Machine Learning Systems , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering.