Requirements Engineering for Machine Learning: Perspectives from Data Scientists

Machine learning (ML) is used increasingly in real-world applications. In this paper, we describe our ongoing endeavor to define characteristics and challenges unique to Requirements Engineering (RE) for ML-based systems. As a first step, we interviewed four data scientists to understand how ML experts approach elicitation, specification, and assurance of requirements and expectations. The results show that changes in the development paradigm, i.e., from coding to training, also demands changes in RE. We conclude that development of ML systems demands requirements engineers to: (1) understand ML performance measures to state good functional requirements, (2) be aware of new quality requirements such as explainability, freedom from discrimination, or specific legal requirements, and (3) integrate ML specifics in the RE process. Our study provides a first contribution towards an RE methodology for ML systems.

[1]  Lior Rokach,et al.  Introduction to Knowledge Discovery and Data Mining , 2010, Data Mining and Knowledge Discovery Handbook.

[2]  Danny Weyns,et al.  A Classification Framework of Uncertainty in Architecture-Based Self-Adaptive Systems with Multiple Quality Requirements , 2015 .

[3]  Takeo Kanade,et al.  Software Engineering for Self-Adaptive Systems II , 2013, Lecture Notes in Computer Science.

[4]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[5]  Jonas Paul Winkler,et al.  Optimizing for Recall in Automatic Requirements Classification: An Empirical Study , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[6]  Rogério de Lemos,et al.  Software Engineering for Self-Adaptive Systems [outcome of a Dagstuhl Seminar] , 2009, Software Engineering for Self-Adaptive Systems.

[7]  Danny Weyns,et al.  Software Engineering of Self-adaptive Systems , 2019, Handbook of Software Engineering.

[8]  Graham R. Gibbs,et al.  Analyzing Qualitative Data , 2014 .

[9]  Trevor Darrell,et al.  Women also Snowboard: Overcoming Bias in Captioning Models , 2018, ECCV.

[10]  Anna Perini,et al.  Engineering requirements for adaptive systems , 2015, Requirements Engineering.

[11]  D. Sculley,et al.  The ML test score: A rubric for ML production readiness and technical debt reduction , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[12]  Anna Perini,et al.  A Machine Learning Approach to Software Requirements Prioritization , 2013, IEEE Transactions on Software Engineering.

[13]  Natalia Miloslavskaya,et al.  Big Data, Fast Data and Data Lake Concepts , 2016, BICA.

[14]  Andreas Vogelsang,et al.  Automatic Classification of Requirements Based on Convolutional Neural Networks , 2016, 2016 IEEE 24th International Requirements Engineering Conference Workshops (REW).

[15]  Eric S. K. Yu,et al.  Business-driven data analytics: A conceptual modeling framework , 2018, Data Knowl. Eng..

[16]  Walid Maalej,et al.  On the automatic classification of app reviews , 2016, Requirements Engineering.

[17]  Jianwu Wang,et al.  Big data provenance: Challenges, state of the art and opportunities , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[18]  Daniel M. Berry,et al.  Evaluation of Tools for Hairy Requirements and Software Engineering Tasks , 2017, 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW).

[19]  Mahshid Helali Moghadam,et al.  Machine Learning to Guide Performance Testing: An Autonomous Test Framework , 2019, 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[20]  Andreas Vogelsang,et al.  Extraction of System States from Natural Language Requirements , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[21]  Foutse Khomh,et al.  Software Engineering for Machine-Learning Applications: The Road Ahead , 2018, IEEE Software.

[22]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[23]  Björn Regnell,et al.  An investigation of how quality requirements are specified in industrial practice , 2013, Inf. Softw. Technol..

[24]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[25]  Markus Borg,et al.  Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry , 2018, Journal of Automotive Software Engineering.

[26]  Fuyuki Ishikawa,et al.  How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey , 2019, 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP).

[27]  Amina Asif,et al.  Ten ways to fool the masses with machine learning , 2019, ArXiv.

[28]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .

[29]  Lin Liu,et al.  Requirements Engineering for Health Data Analytics: Challenges and Possible Directions , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[30]  Jonas Paul Winkler,et al.  "What Does My Classifier Learn?" A Visual Approach to Understanding Natural Language Text Classifiers , 2017, NLDB.

[31]  Jennifer Horkoff,et al.  Non-Functional Requirements for Machine Learning: Challenges and New Directions , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[32]  Mehrdad Sabetzadeh,et al.  An Active Learning Approach for Improving the Accuracy of Automated Domain Model Extraction , 2019, ACM Trans. Softw. Eng. Methodol..

[33]  Yujie Li,et al.  Using Reinforcement Learning to Handle the Runtime Uncertainties in Self-adaptive Software , 2018, STAF Workshops.