Quality Assurance for Machine Learning – an approach to function and system safeguarding

In an industrial context, high software quality is mandatory in order to avoid costly patching. We present a state of the art analysis of approaches to ensure that a specific Artificial Intelligence (AI) model is ready for release. We analyze the requirements a Machine Learning (ML) system has to fulfill in order to comply with the needs of an automotive OEM. The main implication for projects relying on ML is a holistic assessment of possible quality risks. These risks may stem from implemented ML models and spread into the delivery. We present a methodological quality assurance (QA) approach and its evaluation.

[1]  Dong Yu,et al.  Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[2]  Samridhi Jha.  Data Infrastructure for Machine Learning , 2019 .

[3]  Dawn Song,et al.  Robust Physical-World Attacks on Deep Learning Models , 2017, 1707.08945.

[4]  Richard T. Watson,et al.  Analyzing the Past to Prepare for the Future: Writing a Literature Review , 2002, MIS Q..

[5]  Alexander Poth,et al.  Quality Requirements Elicitation by Ideation of Product Quality Risks with Design Thinking , 2020, 2020 IEEE 28th International Requirements Engineering Conference (RE).

[6]  Robert Winter,et al.  Design science research in Europe , 2008 .

[7]  Alexei A. Efros,et al.  Unbiased look at dataset bias , 2011, CVPR 2011.

[8]  Samy Bengio,et al.  Adversarial Machine Learning at Scale , 2016, ICLR.

[9]  Jimmy J. Lin,et al.  Large-scale machine learning at twitter , 2012, SIGMOD Conference.

[10]  Yang Song,et al.  Improving the Robustness of Deep Neural Networks via Stability Training , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  David Yeh.  Autonomous Systems and the Challenges in Verification, Validation, and Test , 2018, IEEE Des. Test.

[12]  Vatche Ishakian,et al.  Serving Deep Learning Models in a Serverless Platform , 2017, 2018 IEEE International Conference on Cloud Engineering (IC2E).

[13]  D. Sculley,et al.  What’s your ML test score? A rubric for ML production systems , 2016 .

[14]  Alexander Poth,et al.  Effective Quality Management : Risk-and Value-based Software Quality Management , 2013 .

[15]  Felix Bießmann,et al.  On Challenges in Machine Learning Model Management , 2018, IEEE Data Eng. Bull..

[16]  Chris Murphy,et al.  An Approach to Software Testing of Machine Learning Applications , 2007, SEKE.

[17]  Alexander Poth,et al.  How to Innovate Software Quality Assurance and Testing in Large Enterprises? , 2018, EuroSPI.

[18]  Tim Menzies,et al.  Verification and Validation and Artificial Intelligence , 2005, Adv. Comput..

[19]  Foster Provost,et al.  Machine Learning from Imbalanced Data Sets 101 , 2008 .

[20]  Alexander Poth,et al.  How to Deliver Faster with CI/CD Integrated Testing Services? , 2018, EuroSPI.

[21]  Christian Kreiner,et al.  Integrated design for tackling safety and security challenges of smart products and digital manufacturing , 2017 .

[22]  D. Sculley,et al.  TensorFlow Debugger: Debugging Dataflow Graphs for Machine Learning , 2016 .

[23]  Alexander Poth,et al.  Scaling Agile - A Large Enterprise View on Delivering and Ensuring Sustainable Transitions , 2019, LADS/MIDI.

[24]  Ali Sunyaev,et al.  Effective Quality Management: Value- and Risk-Based Software Quality Management , 2014, IEEE Software.