Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends

As deep learning continues to dominate all state-of-the-art computer vision tasks, it is increasingly becoming an essential building block for robotic perception. This raises important questions concerning the safety and reliability of learning-based perception systems. There is an established field that studies safety certification and convergence guarantees of complex software systems at design-time. However, the unknown future deployment environments of an autonomous system and the complexity of learning-based perception make the generalization of design-time verification to run-time problematic. In the face of this challenge, more attention is starting to focus on run-time monitoring of performance and reliability of perception systems with several trends emerging in the literature in the face of this challenge. This paper attempts to identify these trends and summarize the various approaches to the topic.

[1]  Niko Sünderhauf,et al.  Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Roland Siegwart,et al.  Out-of-Distribution Detection for Automotive Perception , 2020, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC).

[3]  Ben Glocker,et al.  Real-time Prediction of Segmentation Quality , 2018, MICCAI.

[4]  Tristan Perez,et al.  Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting—Combined Color and 3-D Information , 2017, IEEE Robotics and Automation Letters.

[5]  C. K. Chow,et al.  On optimum recognition error and reject tradeoff , 1970, IEEE Trans. Inf. Theory.

[6]  Michael Milford,et al.  Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[7]  Ingmar Posner,et al.  Wrong Today, Right Tomorrow: Experience-Based Classification for Robot Perception , 2015, FSR.

[8]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[9]  James J. Little,et al.  Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors , 2018, ArXiv.

[10]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[11]  Matti Pietikäinen,et al.  Deep Learning for Generic Object Detection: A Survey , 2018, International Journal of Computer Vision.

[12]  Matthias Rottmann,et al.  MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection , 2020, 2021 International Joint Conference on Neural Networks (IJCNN).

[13]  Charles Richter,et al.  Safe Visual Navigation via Deep Learning and Novelty Detection , 2017, Robotics: Science and Systems.

[14]  Stephan Günnemann,et al.  Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift , 2018, NeurIPS.

[15]  Ran El-Yaniv,et al.  Selective Classification for Deep Neural Networks , 2017, NIPS.

[16]  Thomas G. Dietterich,et al.  A Unifying Review of Deep and Shallow Anomaly Detection , 2020, Proceedings of the IEEE.

[17]  Konstantinos Kamnitsas,et al.  Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth , 2017, IEEE Transactions on Medical Imaging.

[18]  Zhangyang Wang,et al.  Predicting Model Failure using Saliency Maps in Autonomous Driving Systems , 2019, ArXiv.

[19]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[20]  Vladimir Vovk,et al.  A tutorial on conformal prediction , 2007, J. Mach. Learn. Res..

[21]  Sadegh Rabiee,et al.  IV-SLAM: Introspective Vision for Simultaneous Localization and Mapping , 2020, CoRL.

[22]  Rick Salay,et al.  Bayesian Uncertainty Quantification with Synthetic Data , 2019, SAFECOMP Workshops.

[23]  Chih-Hong Cheng,et al.  Runtime Monitoring Neuron Activation Patterns , 2018, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[24]  Yang Tang,et al.  Monocular depth estimation based on deep learning: An overview , 2020, Science China Technological Sciences.

[25]  Fabio Roli,et al.  Support Vector Machines with Embedded Reject Option , 2002, SVM.

[26]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[27]  Luca Carlone,et al.  Monitoring and Diagnosability of Perception Systems , 2020, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[29]  Nuno Vasconcelos,et al.  Towards Realistic Predictors , 2018, ECCV.

[30]  Di Feng,et al.  A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving , 2020, IEEE Transactions on Intelligent Transportation Systems.

[31]  Martial Hebert,et al.  Introspective perception: Learning to predict failures in vision systems , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[32]  Steven L. Waslander,et al.  BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Kostas Daniilidis,et al.  Convolutional Mesh Regression for Single-Image Human Shape Reconstruction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Marta Z. Kwiatkowska,et al.  Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control , 2018, ArXiv.

[35]  Gereon Weiss,et al.  Benchmarking Uncertainty Estimation Methods for Deep Learning With Safety-Related Metrics , 2020, SafeAI@AAAI.

[36]  Ioannis Patras,et al.  Mirror, mirror on the wall, tell me, is the error small? , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Xenofon Koutsoukos,et al.  Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systems , 2020, 2020 ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS).

[38]  Mehryar Mohri,et al.  Learning with Rejection , 2016, ALT.

[39]  Klaus C. J. Dietmayer,et al.  Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[40]  Alex Kendall,et al.  Concrete Dropout , 2017, NIPS.

[41]  Terrance E. Boult,et al.  Automatic Open-World Reliability Assessment , 2020, ArXiv.

[42]  Tristan Perez,et al.  A Rapidly Deployable Classification System using Visual Data for the Application of Precision Weed Management , 2018, Comput. Electron. Agric..

[43]  Daniel Kroening,et al.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability , 2018, Comput. Sci. Rev..

[44]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Ali Farhadi,et al.  Predicting Failures of Vision Systems , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Hanno Gottschalk,et al.  Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities , 2018, 2020 International Joint Conference on Neural Networks (IJCNN).

[47]  Rudolph Triebel,et al.  Introspective classification for robot perception , 2016, Int. J. Robotics Res..

[48]  Paul Newman,et al.  Know your limits: Embedding localiser performance models in teach and repeat maps , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[49]  Martial Hebert,et al.  Learning robust failure response for autonomous vision based flight , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[50]  Feras Dayoub,et al.  Performance Monitoring of Object Detection During Deployment , 2020, ArXiv.

[51]  Mehryar Mohri,et al.  Boosting with Abstention , 2016, NIPS.

[52]  Xiaofeng Liu,et al.  Deep Verifier Networks: Verification of Deep Discriminative Models with Deep Generative Models , 2019, AAAI.

[53]  Matthew Johnson-Roberson,et al.  Failing to Learn: Autonomously Identifying Perception Failures for Self-Driving Cars , 2017, IEEE Robotics and Automation Letters.

[54]  Matthieu Cord,et al.  Addressing Failure Prediction by Learning Model Confidence , 2019, NeurIPS.

[55]  Johann Marius Zöllner,et al.  Calibrating Uncertainty Models for Steering Angle Estimation , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[56]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[57]  Feras Dayoub,et al.  Online Monitoring of Object Detection Performance Post-Deployment , 2020, ArXiv.

[58]  Alexey Dosovitskiy,et al.  End-to-End Driving Via Conditional Imitation Learning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[59]  Luc Van Gool,et al.  Failure Prediction for Autonomous Driving , 2018, 2018 IEEE Intelligent Vehicles Symposium (IV).

[60]  C. V. Jawahar,et al.  Has My Algorithm Succeeded? An Evaluator for Human Pose Estimators , 2012, ECCV.

[61]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[62]  Martin E. Hellman,et al.  The Nearest Neighbor Classification Rule with a Reject Option , 1970, IEEE Trans. Syst. Sci. Cybern..

[63]  Gabriele Costante,et al.  Uncertainty Estimation for Data-Driven Visual Odometry , 2020, IEEE Transactions on Robotics.

[64]  Feras Dayoub,et al.  Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment , 2021, 2021 IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW).

[65]  Shaolei Ren,et al.  Increasing the Trustworthiness of Deep Neural Networks via Accuracy Monitoring , 2020, AISafety@IJCAI.

[66]  Li Liu,et al.  A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges , 2020, Inf. Fusion.

[67]  Dushyant Rao,et al.  Learn from experience: Probabilistic prediction of perception performance to avoid failure , 2018, Int. J. Robotics Res..

[68]  Stewart Worrall,et al.  Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving , 2018, IEEE Transactions on Intelligent Transportation Systems.

[69]  Yen-Cheng Liu,et al.  UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[70]  Winston Churchill,et al.  Off the beaten track: Predicting localisation performance in visual teach and repeat , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[71]  Gordon Wyeth,et al.  Place categorization and semantic mapping on a mobile robot , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[72]  Hanno Gottschalk,et al.  Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks , 2019, 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI).

[73]  Thomas A. Henzinger,et al.  Outside the Box: Abstraction-Based Monitoring of Neural Networks , 2019, ECAI.

[74]  Joost van de Weijer,et al.  Metric Learning for Novelty and Anomaly Detection , 2018, BMVC.

[75]  Joydeep Biswas,et al.  IVOA: Introspective Vision for Obstacle Avoidance , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[76]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[77]  C. K. Chow,et al.  An optimum character recognition system using decision functions , 1957, IRE Trans. Electron. Comput..

[78]  Benjamin Recht,et al.  Do ImageNet Classifiers Generalize to ImageNet? , 2019, ICML.

[79]  Luca Carlone,et al.  Online Monitoring for Neural Network Based Monocular Pedestrian Pose Estimation , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).