Revisiting Batch Normalization for Improving Corruption Robustness

The performance of DNNs trained on clean images has been shown to decrease when the test images have common corruptions. In this work, we interpret corruption robustness as a domain shift and propose to rectify batch normalization (BN) statistics for improving model robustness. This is motivated by perceiving the shift from the clean domain to the corruption domain as a style shift that is represented by the BN statistics. We find that simply estimating and adapting the BN statistics on a few (32 for instance) representation samples, without retraining the model, improves the corruption robustness by a large margin on several benchmark datasets with a wide range of model architectures. For example, on ImageNet-C, statistics adaptation improves the top1 accuracy of ResNet50 from 39.2% to 48.7%. Moreover, we find that this technique can further improve state-of-the-art robust models from 58.1% to 63.3%.

[1]  In So Kweon,et al.  Double Targeted Universal Adversarial Perturbations , 2020, ACCV.

[2]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Frank Wang,et al.  The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  Matthias Bethge,et al.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness , 2018, ICLR.

[5]  Alan L. Yuille,et al.  Intriguing Properties of Adversarial Training at Scale , 2020, ICLR.

[6]  Bin Chen,et al.  Targeted Attack for Deep Hashing based Retrieval , 2020, ECCV.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[9]  Nic Ford,et al.  Adversarial Examples Are a Natural Consequence of Test Error in Noise , 2019, ICML.

[10]  Hal Daumé,et al.  Frustratingly Easy Domain Adaptation , 2007, ACL.

[11]  Aleksander Madry,et al.  How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) , 2018, NeurIPS.

[12]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[13]  Trevor Darrell,et al.  Adversarial Discriminative Domain Adaptation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  In So Kweon,et al.  CD-UAP: Class Discriminative Universal Adversarial Perturbation , 2020, AAAI.

[15]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[16]  In-So Kweon,et al.  Revisiting Residual Networks with Nonlinear Shortcuts , 2019, BMVC.

[17]  In So Kweon,et al.  Batch Normalization Increases Adversarial Vulnerability: Disentangling Usefulness and Robustness of Model Features , 2020, ArXiv.

[18]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[19]  In So Kweon,et al.  Recurrent Temporal Aggregation Framework for Deep Video Inpainting , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Myungchul Kim,et al.  The Devil is in the Boundary: Exploiting Boundary Representation for Basis-based Instance Segmentation , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[21]  David A. Wagner,et al.  Towards Evaluating the Robustness of Neural Networks , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[22]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[23]  Ping Luo,et al.  Towards Understanding Regularization in Batch Normalization , 2018, ICLR.

[24]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  In So Kweon,et al.  Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Bin Chen,et al.  Adversarial Attack on Deep Product Quantization Network for Image Retrieval , 2020, AAAI.

[28]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Chaoning Zhang,et al.  Understanding Adversarial Examples From the Mutual Influence of Images and Perturbations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[31]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[33]  Ekin D. Cubuk,et al.  A Fourier Perspective on Model Robustness in Computer Vision , 2019, NeurIPS.

[34]  Taesung Park,et al.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation , 2017, ICML.

[35]  Carsten Rother,et al.  Benchmarking the Robustness of Semantic Segmentation Models with Respect to Common Corruptions , 2019, International Journal of Computer Vision.

[36]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Balaji Lakshminarayanan,et al.  AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty , 2020, ICLR.

[38]  Muhammad Awais,et al.  Revisiting Internal Covariate Shift for Batch Normalization , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Sung-Ho Bae,et al.  Towards an Adversarially Robust Normalization Approach , 2019, ArXiv.

[40]  Jiaying Liu,et al.  Revisiting Batch Normalization For Practical Domain Adaptation , 2016, ICLR.

[41]  Koby Crammer,et al.  Analysis of Representations for Domain Adaptation , 2006, NIPS.

[42]  Hyo-Eun Kim,et al.  Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks , 2018, NeurIPS.

[43]  Thomas G. Dietterich,et al.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , 2018, ICLR.

[44]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[45]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[46]  Ming Yang,et al.  Conditional Generative Adversarial Network for Structured Domain Adaptation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[48]  Alexander S. Ecker,et al.  Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming , 2019, ArXiv.

[49]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Chaoning Zhang,et al.  DeepPTZ: Deep Self-Calibration for PTZ Cameras , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[51]  Chaoning Zhang,et al.  ResNet or DenseNet? Introducing Dense Shortcuts to ResNet , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[52]  In So Kweon,et al.  Video Panoptic Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  In So Kweon,et al.  Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracy , 2020, ArXiv.

[54]  Gijs Dubbelman,et al.  A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[55]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[56]  Kate Saenko,et al.  Correlation Alignment for Unsupervised Domain Adaptation , 2016, Domain Adaptation in Computer Vision Applications.

[57]  Chaoning Zhang,et al.  Data from Model: Extracting Data from Non-robust and Robust Models , 2020, ArXiv.

[58]  Fabio Maria Carlucci,et al.  AutoDIAL: Automatic Domain Alignment Layers , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[59]  Thomas Hofmann,et al.  Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization , 2018, AISTATS.

[60]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.