FSNet: A Failure Detection Framework for Semantic Segmentation

Semantic segmentation is an important task that helps autonomous vehicles understand their surroundings and navigate safely. However, during deployment, even the most mature segmentation models are vulnerable to various external factors that can degrade the segmentation performance with potentially catastrophic consequences for the vehicle and its surroundings. To address this issue, we propose a failure detection framework to identify pixel-level misclassification. We do so by exploiting internal features of the segmentation model and training it simultaneously with a failure detection network. During deployment, the failure detector flags areas in the image where the segmentation model has failed to segment correctly. We evaluate the proposed approach against state-ofthe-art methods and achieve 12.30%, 9.46%, and 9.65% performance improvement in the AUPR-Error metric for Cityscapes, BDD100k, and Mapillary semantic segmentation datasets.

[1]  Hélène Laurent,et al.  Unsupervised Performance Evaluation of Image Segmentation , 2006, EURASIP J. Adv. Signal Process..

[2]  Rudolph Triebel,et al.  Driven Learning for Driving: How Introspection Improves Semantic Mapping , 2016, ISRR.

[3]  Dushyant Rao,et al.  Learn from experience: Probabilistic prediction of perception performance to avoid failure , 2018, Int. J. Robotics Res..

[4]  Shiguo Lian,et al.  Small Obstacle Avoidance Based on RGB-D Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[5]  José García Rodríguez,et al.  A survey on deep learning techniques for image and video semantic segmentation , 2018, Appl. Soft Comput..

[6]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Ran El-Yaniv,et al.  Selective Classification for Deep Neural Networks , 2017, NIPS.

[8]  Thomas G. Dietterich,et al.  A Unifying Review of Deep and Shallow Anomaly Detection , 2020, Proceedings of the IEEE.

[9]  Alexander Rakhlin,et al.  Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning , 2018, bioRxiv.

[10]  Haiqiang Chen,et al.  Road segmentation for all-day outdoor robot navigation , 2018, Neurocomputing.

[11]  Timo Kohlberger,et al.  Evaluating Segmentation Error without Ground Truth , 2012, MICCAI.

[12]  Eckehard Steinbach,et al.  Introspective Failure Prediction for Semantic Image Segmentation , 2020, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC).

[13]  William Whittaker,et al.  Robotic introspection for exploration and mapping of subterranean environments , 2007 .

[14]  Ali Farhadi,et al.  Predicting Failures of Vision Systems , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Klaus C. J. Dietmayer,et al.  Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges , 2019, IEEE Transactions on Intelligent Transportation Systems.

[16]  Chao Huang,et al.  QualityNet: Segmentation quality evaluation with deep convolutional networks , 2016, 2016 Visual Communications and Image Processing (VCIP).

[17]  Yanrong Guo,et al.  A Brief Survey on Semantic Segmentation with Deep Learning , 2020, Neurocomputing.

[18]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[19]  Daniel Cagara,et al.  Improving Underwater Obstacle Detection using Semantic Image Segmentation , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[20]  Graham W. Taylor,et al.  Leveraging Uncertainty Estimates for Predicting Segmentation Quality , 2018, ArXiv.

[21]  Matthieu Cord,et al.  Addressing Failure Prediction by Learning Model Confidence , 2019, NeurIPS.

[22]  Yassine Ruichek,et al.  Survey on semantic segmentation using deep learning techniques , 2019, Neurocomputing.

[23]  Kun Yu,et al.  DenseASPP for Semantic Segmentation in Street Scenes , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Peter Kontschieder,et al.  The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Lena Maier-Hein,et al.  2017 Robotic Instrument Segmentation Challenge , 2019, ArXiv.

[26]  Peter Corke,et al.  Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends , 2021, IEEE Access.

[27]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Sepp Hochreiter,et al.  Speeding up Semantic Segmentation for Autonomous Driving , 2016 .

[29]  Ben Glocker,et al.  Real-time Prediction of Segmentation Quality , 2018, MICCAI.

[30]  Sebastian Scherer,et al.  Real-Time Semantic Mapping for Autonomous Off-Road Navigation , 2017, FSR.

[31]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Rudolph Triebel,et al.  Introspective classification for robot perception , 2016, Int. J. Robotics Res..

[33]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[34]  Antonio J. Plaza,et al.  Image Segmentation Using Deep Learning: A Survey , 2021, IEEE transactions on pattern analysis and machine intelligence.

[35]  Hui Li,et al.  A Novel Unsupervised Segmentation Quality Evaluation Method for Remote Sensing Images , 2017, Sensors.

[36]  Martial Hebert,et al.  Introspective perception: Learning to predict failures in vision systems , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[37]  Nuno Vasconcelos,et al.  Towards Realistic Predictors , 2018, ECCV.

[38]  Maya R. Gupta,et al.  To Trust Or Not To Trust A Classifier , 2018, NeurIPS.

[39]  Yingda Xia,et al.  Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation , 2020, ECCV.

[40]  Songcan Chen,et al.  Recent Advances in Open Set Recognition: A Survey , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Mauricio Reyes,et al.  Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation , 2018, ArXiv.

[42]  Konstantinos Kamnitsas,et al.  Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth , 2017, IEEE Transactions on Medical Imaging.