Hardening of Artificial Neural Networks for Use in Safety-Critical Applications - A Mapping Study

Context: Across different domains, Artificial Neural Networks (ANNs) are used more and more in safety-critical applications in which erroneous outputs of such ANN can have catastrophic consequences. However, the development of such neural networks is still immature and good engineering practices are missing. With that, ANNs are in the same position as software was several decades ago. Today, standards for functional safety, such as ISO 26262 in the automotive domain, require the application of a collection of proven engineering principles and methods in the creation of software to increase its quality and reduce failure rates to an acceptable level. Objective: In the future, such a set of proven engineering methods needs to be established for the development of Artificial Neural Networks to allow their use in safety-critical applications. Method: This work takes a step in this direction by conducting a mapping study to extract challenges faced in the development of ANNs for safety-critical applications and to identify methods that have been used for the hardening of ANNs in such settings. Results: We extracted ten different challenges found to be repeatedly reported in the literature regarding the use of ANNs in critical contexts. All of these challenges are addressed by engineering methods, of which we identified 54 in our study that can be used for the hardening of networks. Conclusions: Various methods have been proposed to overcome the specific challenges of using ANNs in safety-critical applications. On the path towards defining best practices, we envision that future software engineering will need to focus on further investigating these methods and increasing the maturity and understanding of existing approaches, with the goal to develop clear guidance for proper engineering of high-quality ANNs.

[1]  Chih-Hong Cheng,et al.  Neural networks for safety-critical applications — Challenges, experiments and perspectives , 2017, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[2]  R.R. Zakrzewski,et al.  Randomized approach to verification of neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[3]  Xiang-dong Chen,et al.  Multisensor Track Occupancy Detection Model Based on Chaotic Neural Networks , 2015, Int. J. Distributed Sens. Networks.

[4]  Chaulwar Amit,et al.  A Hybrid Machine Learning Approach for Planning Safe Trajectories in Complex Traffic-Scenarios , 2016 .

[5]  Homayoun Seraji,et al.  Rule-based reasoning and neural network perception for safe off-road robot mobility , 2002, Expert Syst. J. Knowl. Eng..

[6]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[7]  Qichao Zhang,et al.  Model-Free Optimal Control Based Intelligent Cruise Control with Hardware-in-the-Loop Demonstration [Research Frontier] , 2017, IEEE Computational Intelligence Magazine.

[8]  Gustavo Pessin,et al.  A Deep Learning Approach to Detect Distracted Drivers Using a Mobile Phone , 2017, ICANN.

[9]  Roberto Cipolla,et al.  Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning , 2017, IJCAI 2017.

[10]  Jim Austin,et al.  Exploiting Safety Constraints in Fuzzy Self-organising Maps for Safety Critical Applications , 2004, IDEAL.

[11]  Canyong Wang,et al.  Research and Application of Traffic Sign Detection and Recognition Based on Deep Learning , 2018, 2018 International Conference on Robots & Intelligent System (ICRIS).

[12]  D. Howard,et al.  Artificial Neural Network Prediction Using Accelerometers to Control Upper Limb FES During Reaching and Grasping Following Stroke , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[13]  Lincheng Shen,et al.  A saliency-based reinforcement learning approach for a UAV to avoid flying obstacles , 2018, Robotics Auton. Syst..

[14]  Simon Burton,et al.  Making the Case for Safety of Machine Learning in Highly Automated Driving , 2017, SAFECOMP Workshops.

[15]  Chaomin Luo,et al.  A computationally efficient neural dynamics approach to trajectory planning of an intelligent vehicle , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[16]  Martin Jägersand,et al.  Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges , 2017, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

[17]  Patrick Doherty,et al.  Deep Learning Quadcopter Control via Risk-Aware Active Learning , 2017, AAAI.

[18]  S. Huseinbegovic,et al.  Road Type Recognition Using Neural Networks for Vehicle Seat Vibration Damping , 2008, 2008 IEEE International Symposium on Signal Processing and Information Technology.

[19]  Marco Botta,et al.  Non-intrusive Detection of Driver Distraction using Machine Learning Algorithms , 2010, ECAI.

[20]  Marcello R. Napolitano,et al.  Bounding set calculation for neural network-based output feedback adaptive control systems , 2010, Neural Computing and Applications.

[21]  Luca Pulina,et al.  An Abstraction-Refinement Approach to Verification of Artificial Neural Networks , 2010, CAV.

[22]  Zhenglong Sun,et al.  Passive magnetic-based localization for precise untethered medical instrument tracking , 2018, Comput. Methods Programs Biomed..

[23]  Wenhu Qin,et al.  Research on a DSRC-Based Rear-End Collision Warning Model , 2014, IEEE Transactions on Intelligent Transportation Systems.

[24]  Kai Petersen,et al.  Guidelines for conducting systematic mapping studies in software engineering: An update , 2015, Inf. Softw. Technol..

[25]  Yang Lei,et al.  Railway Subgrade Defect Automatic Recognition Method Based on Improved Faster R-CNN , 2018, Sci. Program..

[26]  Ammar Belatreche,et al.  An experimental evaluation of novelty detection methods , 2014, Neurocomputing.

[27]  Yan Liu,et al.  Application of Neural Networks in High Assurance Systems: A Survey , 2010, Applications of Neural Networks in High Assurance Systems.

[28]  Xiao Zeng,et al.  MobileDeepPill: A Small-Footprint Mobile Deep Learning System for Recognizing Unconstrained Pill Images , 2017, MobiSys.

[29]  Boubekeur Mendil,et al.  Toward safety navigation in cluttered dynamic environment: A robot neural-based hybrid autonomous navigation and obstacle avoidance with moving target tracking , 2015, 2015 3rd International Conference on Control, Engineering & Information Technology (CEIT).

[30]  Chaomin Luo,et al.  Safety aware robot coverage motion planning with virtual-obstacle-based navigation , 2015, 2015 IEEE International Conference on Information and Automation.

[31]  Brian J. Taylor,et al.  Verification and validation of neural networks: a sampling of research in progress , 2003, SPIE Defense + Commercial Sensing.

[32]  Bojan Cukic,et al.  An Approach to V&V of Embedded Adaptive Systems , 2004, FAABS.

[33]  Liyan Zhang,et al.  Application of Internet of Things Technology and Convolutional Neural Network Model in Bridge Crack Detection , 2018, IEEE Access.

[34]  James H. Graham,et al.  A neuro-fuzzy approach for robot system safety , 2001, IEEE Trans. Syst. Man Cybern. Syst..

[35]  Simon Burton,et al.  Structuring Validation Targets of a Machine Learning Function Applied to Automated Driving , 2018, SAFECOMP.

[36]  ZeFeng Wang,et al.  Aircraft fault diagnosis and decision system based on improved artificial neural networks , 2012, 2012 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[37]  David Lowe,et al.  Point-Wise Confidence Interval Estimation by Neural Networks: A Comparative Study based on Automotive Engine Calibration , 1999, Neural Computing & Applications.

[38]  Wei-Ngan Chin,et al.  Automated Technology for Verification and Analysis , 2010, Lecture Notes in Computer Science.

[39]  Guanpeng Li,et al.  Understanding Error Propagation in Deep Learning Neural Network (DNN) Accelerators and Applications , 2017, SC17: International Conference for High Performance Computing, Networking, Storage and Analysis.

[40]  Chun-Ming Tsai,et al.  Pedestrian, bike, motorcycle, and vehicle classification via deep learning: Deep belief network and small training set , 2016, 2016 International Conference on Applied System Innovation (ICASI).

[41]  Chris J. Harris,et al.  Multi-sensor data fusion for helicopter guidance using neuro-fuzzy estimation algorithms , 1996 .

[42]  Biao Yang,et al.  On Road Vehicle Detection Using an Improved Faster RCNN Framework with Small-Size Region Up-Scaling Strategy , 2017, PSIVT Workshops.

[43]  R. R. Zakrzewski.  Verification of performance of a neural network estimator , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[44]  Thomas A. Runkler,et al.  Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies , 2016, Eng. Appl. Artif. Intell..

[45]  Giuseppe Lami,et al.  Challenges in Certification of Autonomous Driving Systems , 2017, 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW).

[46]  Luca Pulina,et al.  NeVer: a tool for artificial neural networks verification , 2011, Annals of Mathematics and Artificial Intelligence.