Improving MC-Dropout Uncertainty Estimates with Calibration Error-based Optimization

Uncertainty quantification of machine learning and deep learning methods plays an important role in enhancing trust to the obtained result. In recent years, a numerous number of uncertainty quantification methods have been introduced. Monte Carlo dropout (MC-Dropout) is one of the most wellknown techniques to quantify uncertainty in deep learning methods. In this study, we propose two new loss functions by combining cross entropy with Expected Calibration Error (ECE) and Predictive Entropy (PE). The obtained results clearly show that the new proposed loss functions lead to having a calibrated MC-Dropout method. Our results confirmed the great impact of the new hybrid loss functions for minimising the overlap between the distributions of uncertainty estimates for correct and incorrect predictions without sacrificing the model’s overall performance.

[1]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[2]  Saeid Nahavandi,et al.  An Uncertainty-Aware Transfer Learning-Based Framework for COVID-19 Diagnosis , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Dorothea Kolossa,et al.  Detecting Adversarial Examples for Speech Recognition via Uncertainty Quantification , 2020, INTERSPEECH.

[4]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[5]  Adam Kortylewski,et al.  Informed MCMC with Bayesian Neural Networks for Facial Image Analysis , 2018, ArXiv.

[6]  Poulami Sinhamahapatra,et al.  Is Uncertainty Quantification in Deep Learning Sufficient for Out-of-Distribution Detection? , 2020, AISafety@IJCAI.

[7]  Vincent Andrearczyk,et al.  An exploration of uncertainty information for segmentation quality assessment , 2020, Medical Imaging: Image Processing.

[8]  Saeid Nahavandi,et al.  Uncertainty-Aware Credit Card Fraud Detection Using Deep Learning , 2021, Eng. Appl. Artif. Intell..

[9]  Li Liu,et al.  A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges , 2020, Inf. Fusion.

[10]  Maarten De Vos,et al.  SleepTransformer: Automatic Sleep Staging With Interpretability and Uncertainty Quantification , 2021, IEEE Transactions on Biomedical Engineering.

[11]  Morteza Mardani,et al.  Uncertainty Quantification in Deep MRI Reconstruction , 2021, IEEE Transactions on Medical Imaging.

[12]  Saeid Nahavandi,et al.  Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications , 2018, IEEE Access.

[13]  Xiaohan Chen,et al.  Uncertainty Quantification for Deep Context-Aware Mobile Activity Recognition and Unknown Context Discovery , 2020, AISTATS.

[14]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[15]  Saeid Nahavandi,et al.  Objective evaluation of deep uncertainty predictions for COVID-19 detection , 2020, Scientific Reports.

[16]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[17]  Clemens Elster,et al.  Uncertainty quantification by ensemble learning for computational optical form measurements , 2021, Mach. Learn. Sci. Technol..

[18]  Sebastian Nowozin,et al.  Deep Directional Statistics: Pose Estimation with Uncertainty Quantification , 2018, ECCV.

[19]  Wenshi Chen,et al.  Uncertainty quantification for multilabel text classification , 2020, Wiley Interdiscip. Rev. Data Min. Knowl. Discov..

[20]  Farhad Pourpanah,et al.  Dual VAEGAN: A generative model for generalized zero-shot learning , 2021, Applied Soft Computing.

[21]  Matias Valdenegro-Toro,et al.  I Find Your Lack of Uncertainty in Computer Vision Disturbing , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Anuj Sharma,et al.  A Vision-based System for Traffic Anomaly Detection using Deep Learning and Decision Trees , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[23]  Emre Barut,et al.  Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks , 2021, AAAI.

[24]  Frederic Coulon,et al.  Predicting uncertainty of machine learning models for modelling nitrate pollution of groundwater using quantile regression and UNEEC methods. , 2019, The Science of the total environment.

[25]  Antonio Criminisi,et al.  Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution , 2017, MICCAI.

[26]  Alireza Shadman,et al.  An ANN-based ensemble model for change point estimation in control charts , 2021, Appl. Soft Comput..

[27]  Jasper Snoek,et al.  Second opinion needed: communicating uncertainty in medical machine learning , 2021, npj Digital Medicine.

[28]  Saeid Nahavandi,et al.  Uncertainty quantification in skin cancer classification using three-way decision-based Bayesian deep learning , 2021, Comput. Biol. Medicine.

[29]  Mohammad Ehsan Basiri,et al.  An improved evidence-based aggregation method for sentiment analysis , 2019, J. Inf. Sci..

[30]  S. Chib,et al.  Understanding the Metropolis-Hastings Algorithm , 1995 .

[31]  Saeid Nahavandi,et al.  Automated Detection of Presymptomatic Conditions in Spinocerebellar Ataxia Type 2 Using Monte Carlo Dropout and Deep Neural Network Techniques with Electrooculogram Signals , 2020, Sensors.

[32]  Shuang Yu,et al.  Uncertainty-aware domain alignment for anatomical structure segmentation , 2020, Medical Image Anal..

[33]  By W. R. GILKSt,et al.  Adaptive Rejection Sampling for Gibbs Sampling , 2010 .

[34]  Saeid Nahavandi,et al.  Confidence Aware Neural Networks for Skin Cancer Detection , 2021, ArXiv.

[35]  Zhanxing Zhu,et al.  Functional Bayesian Neural Networks for Model Uncertainty Quantification , 2018 .

[36]  Ahmad O. Aseeri.  Uncertainty-Aware Deep Learning-Based Cardiac Arrhythmias Classification Model of Electrocardiogram Signals , 2021, Comput..

[37]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[38]  Saeid Nahavandi,et al.  Partial Adversarial Training for Neural Network-Based Uncertainty Quantification , 2021, IEEE Transactions on Emerging Topics in Computational Intelligence.

[39]  Regina Barzilay,et al.  Uncertainty Quantification Using Neural Networks for Molecular Property Prediction , 2020, J. Chem. Inf. Model..

[40]  Alan Dorin,et al.  Towards Computer Vision and Deep Learning Facilitated Pollination Monitoring for Agriculture , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[41]  Lena Maier-Hein,et al.  Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy , 2017, IEEE Transactions on Biomedical Engineering.

[42]  Mohammad Ehsan Basiri,et al.  Identifying High-Quality User Replies Using Deep Neural Networks , 2021, 2021 7th International Conference on Web Research (ICWR).

[43]  Shaukat Ali,et al.  Prediction Surface Uncertainty Quantification in Object Detection Models for Autonomous Driving , 2021, 2021 IEEE International Conference on Artificial Intelligence Testing (AITest).

[44]  Ewa Szczurek,et al.  ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning , 2019, Scientific Reports.

[45]  Fakhri Karray,et al.  UncertaintyFuseNet: Robust Uncertainty-aware Hierarchical Feature Fusion with Ensemble Monte Carlo Dropout for COVID-19 Detection , 2021, ArXiv.

[46]  W. Gilks,et al.  Adaptive Rejection Metropolis Sampling Within Gibbs Sampling , 1995 .

[47]  Veronika Rocková,et al.  Uncertainty Quantification for Sparse Deep Learning , 2020, AISTATS.

[48]  Se Yoon Lee.  Gibbs sampler and coordinate ascent variational inference: A set-theoretical review , 2020 .

[49]  Max Welling,et al.  Simple and Accurate Uncertainty Quantification from Bias-Variance Decomposition , 2020, ArXiv.