Scalable Software Testing and Verification for Industrial-Scale Systems: The Challenges

In this position paper, we argue that more collaborative research is needed to increase the use of research-led verification and testing techniques in industrialscale projects. We focus on the a) practical applicability and scalability of verification and testing techniques in industrial projects, and b) to autonomous systems. We identify the challenges involved and bring forward some initial suggestions.

[1]  Atul Prakash,et al.  Robust Physical-World Attacks on Machine Learning Models , 2017, ArXiv.

[2]  Antonia Bertolino.  The (Im)maturity level of software testing , 2004, SOEN.

[3]  Tony Gorschek,et al.  A Model for Technology Transfer in Practice , 2006, IEEE Software.

[4]  Loe M. G. Feijs,et al.  Test Generation for Intelligent Networks Using Model Checking , 1997, TACAS.

[6]  Elaine J. Weyuker,et al.  Automated test generation using model checking: an industrial evaluation , 2014, International Journal on Software Tools for Technology Transfer.

[7]  Stefan Edelkamp,et al.  Directed explicit model checking with HSF-SPIN , 2001, SPIN '01.

[8]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[9]  Vahid Garousi,et al.  Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review , 2016, Inf. Softw. Technol..

[10]  WotawaFranz,et al.  Testing with model checkers: a survey , 2009 .

[11]  John R. Callahan,et al.  Automated Software Testing Using Model-Checking , 1996 .

[12]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[14]  Kai Petersen,et al.  SERP-test: a taxonomy for supporting industry–academia communication , 2017, Software Quality Journal.

[15]  Gordon Fraser,et al.  Issues in using model checkers for test case generation , 2009, J. Syst. Softw..

[16]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[17]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[18]  Danny Weyns,et al.  A study and comparison of industrial vs. academic software product line research published at SPLC , 2018, SPLC.

[19]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[20]  Vahid Garousi,et al.  What industry wants from academia in software testing?: Hearing practitioners' opinions , 2017, EASE.

[21]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[22]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[23]  Armin Biere,et al.  Bounded model checking , 2003, Adv. Comput..

[24]  Anila Mjeda.  Standard-compliant testing for safety-related automotive software , 2013 .

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[27]  Samy Bengio,et al.  Adversarial examples in the physical world , 2016, ICLR.

[28]  Satoshi Masuda.  Software Testing in Industry and Academia: A View of Both Sides in Japan , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[29]  Gordon Fraser,et al.  Testing with model checkers: a survey , 2009, Softw. Test. Verification Reliab..

[30]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[31]  Anila Mjeda,et al.  Scalable software testing and verification for industrial-scale systems: the challenges , 2019 .