Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

As deep learning continues to dominate all state-of-the-art tasks in computer vision, it is increasingly becoming the essential building blocks for robotic perception. As a result, the research questions concerning the safety and reliability of learning-based perception during run-time are gaining increased importance. Although there is an established field that studies safety certification and convergence guarantee of complex software systems during design-time, the uncertainty in the run-time conditions and the unknown deployment environments of autonomous systems and the complexity of learning-based perception systems makes the generalisation of the verification results from design-time to run-time problematic. In the face of such a challenge, more attention is starting to shift towards run-time monitoring of performance and reliability with a number of trends emerging in the literature – this paper attempt to identify these trends and provide a summary of the various approaches on the topic.

[1]  Niko Sünderhauf,et al.  Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Michael Milford,et al.  Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[3]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[4]  James J. Little,et al.  Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors , 2018, ArXiv.

[5]  Marta Z. Kwiatkowska,et al.  Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control , 2018, ArXiv.

[6]  C. V. Jawahar,et al.  Has My Algorithm Succeeded? An Evaluator for Human Pose Estimators , 2012, ECCV.

[7]  Tristan Perez,et al.  Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting—Combined Color and 3-D Information , 2017, IEEE Robotics and Automation Letters.

[8]  Fabio Roli,et al.  Support Vector Machines with Embedded Reject Option , 2002, SVM.

[9]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[10]  Nuno Vasconcelos,et al.  Towards Realistic Predictors , 2018, ECCV.

[11]  Johann Marius Zöllner,et al.  Calibrating Uncertainty Models for Steering Angle Estimation , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[12]  Kostas Daniilidis,et al.  Convolutional Mesh Regression for Single-Image Human Shape Reconstruction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[14]  Feras Dayoub,et al.  Performance Monitoring of Object Detection During Deployment , 2020, ArXiv.

[15]  Daniel Kroening,et al.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability , 2018, Comput. Sci. Rev..

[16]  Ioannis Patras,et al.  Mirror, mirror on the wall, tell me, is the error small? , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[18]  Klaus C. J. Dietmayer,et al.  Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[19]  Thomas A. Henzinger,et al.  Outside the Box: Abstraction-Based Monitoring of Neural Networks , 2019, ECAI.

[20]  Hanno Gottschalk,et al.  Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks , 2019, 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI).

[21]  Ben Glocker,et al.  Real-time Prediction of Segmentation Quality , 2018, MICCAI.

[22]  Vladimir Vovk,et al.  A tutorial on conformal prediction , 2007, J. Mach. Learn. Res..

[23]  Gabriele Costante,et al.  Uncertainty Estimation for Data-Driven Visual Odometry , 2020, IEEE Transactions on Robotics.

[24]  Joydeep Biswas,et al.  IVOA: Introspective Vision for Obstacle Avoidance , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[25]  Di Feng,et al.  A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving , 2020, ArXiv.

[26]  Konstantinos Kamnitsas,et al.  Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth , 2017, IEEE Transactions on Medical Imaging.

[27]  Shaolei Ren,et al.  Increasing the Trustworthiness of Deep Neural Networks via Accuracy Monitoring , 2020, AISafety@IJCAI.

[28]  Zhangyang Wang,et al.  Predicting Model Failure using Saliency Maps in Autonomous Driving Systems , 2019, ArXiv.

[29]  Luc Van Gool,et al.  Failure Prediction for Autonomous Driving , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[30]  Dushyant Rao,et al.  Learn from experience: Probabilistic prediction of perception performance to avoid failure , 2018, Int. J. Robotics Res..

[31]  Paul Newman,et al.  Know your limits: Embedding localiser performance models in teach and repeat maps , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Terrance E. Boult,et al.  Automatic Open-World Reliability Assessment , 2020, ArXiv.

[33]  Luca Carlone,et al.  Online Monitoring for Neural Network Based Monocular Pedestrian Pose Estimation , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[34]  C. K. Chow,et al.  On optimum recognition error and reject tradeoff , 1970, IEEE Trans. Inf. Theory.

[35]  Chih-Hong Cheng,et al.  Runtime Monitoring Neuron Activation Patterns , 2018, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[36]  Tristan Perez,et al.  A Rapidly Deployable Classification System using Visual Data for the Application of Precision Weed Management , 2018, Comput. Electron. Agric..

[37]  Yen-Cheng Liu,et al.  UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[38]  Stephan Günnemann,et al.  Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift , 2018, NeurIPS.

[39]  Xenofon Koutsoukos,et al.  Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systems , 2020, 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS).

[40]  Martial Hebert,et al.  Introspective perception: Learning to predict failures in vision systems , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  Winston Churchill,et al.  Off the beaten track: Predicting localisation performance in visual teach and repeat , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[42]  C. K. Chow,et al.  An optimum character recognition system using decision functions , 1957, IRE Trans. Electron. Comput..

[43]  Yang Tang,et al.  Monocular depth estimation based on deep learning: An overview , 2020, ArXiv.

[44]  Matti Pietikäinen,et al.  Deep Learning for Generic Object Detection: A Survey , 2018, International Journal of Computer Vision.

[45]  Charles Richter,et al.  Safe Visual Navigation via Deep Learning and Novelty Detection , 2017, Robotics: Science and Systems.

[46]  Hanno Gottschalk,et al.  Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities , 2018, 2020 International Joint Conference on Neural Networks (IJCNN).

[47]  Rick Salay,et al.  Bayesian Uncertainty Quantification with Synthetic Data , 2019, SAFECOMP Workshops.

[48]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Gordon Wyeth,et al.  Place categorization and semantic mapping on a mobile robot , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[50]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[51]  Rudolph Triebel,et al.  Introspective classification for robot perception , 2016, Int. J. Robotics Res..

[52]  Li Liu,et al.  A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges , 2020, Inf. Fusion.

[53]  Gereon Weiss,et al.  Benchmarking Uncertainty Estimation Methods for Deep Learning With Safety-Related Metrics , 2020, SafeAI@AAAI.

[54]  Steven L. Waslander,et al.  BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[55]  Ali Farhadi,et al.  Predicting Failures of Vision Systems , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Martial Hebert,et al.  Learning robust failure response for autonomous vision based flight , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[57]  Benjamin Recht,et al.  Do ImageNet Classifiers Generalize to ImageNet? , 2019, ICML.

[58]  Luca Carlone,et al.  Monitoring and Diagnosability of Perception Systems , 2020, ArXiv.

[59]  Xiaofeng Liu,et al.  Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models , 2019, AAAI.

[60]  Matthew Johnson-Roberson,et al.  Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars , 2017, IEEE Robotics and Automation Letters.

[61]  Alexey Dosovitskiy,et al.  End-to-End Driving Via Conditional Imitation Learning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[62]  Sadegh Rabiee,et al.  IV-SLAM: Introspective Vision for Simultaneous Localization and Mapping , 2020, CoRL.

[63]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[64]  Roland Siegwart,et al.  Out-of-Distribution Detection for Automotive Perception , 2020, ArXiv.

[65]  Ran El-Yaniv,et al.  Selective Classification for Deep Neural Networks , 2017, NIPS.

[66]  Stewart Worrall,et al.  Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving , 2018, IEEE Transactions on Intelligent Transportation Systems.

[67]  Matthieu Cord,et al.  Addressing Failure Prediction by Learning Model Confidence , 2019, NeurIPS.

[68]  Ingmar Posner,et al.  Wrong Today, Right Tomorrow: Experience-Based Classification for Robot Perception , 2015, FSR.

[69]  Alex Kendall,et al.  Concrete Dropout , 2017, NIPS.

[70]  Joost van de Weijer,et al.  Metric Learning for Novelty and Anomaly Detection , 2018, BMVC.

[71]  Feras Dayoub,et al.  Online Monitoring of Object Detection Performance Post-Deployment , 2020, ArXiv.

[72]  Martin E. Hellman,et al.  The Nearest Neighbor Classification Rule with a Reject Option , 1970, IEEE Trans. Syst. Sci. Cybern..

[73]  Mehryar Mohri,et al.  Boosting with Abstention , 2016, NIPS.

[74]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[75]  Thomas G. Dietterich,et al.  A Unifying Review of Deep and Shallow Anomaly Detection , 2020, Proceedings of the IEEE.

[76]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[77]  Mehryar Mohri,et al.  Learning with Rejection , 2016, ALT.

[78]  Matthias Rottmann,et al.  MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection , 2020, 2021 International Joint Conference on Neural Networks (IJCNN).