Clustering-Based Adaptive Dropout for CNN-Based Classification

Dropout has been widely used to improve the generalization ability of a deep network, while current dropout variants rarely adapt the dropout probabilities of the network hidden units or weights dynamically to their contributions on the network optimization. In this work, a clustering-based dropout based on the network characteristics of features, weights or their derivatives is proposed, where the dropout probabilities for these characteristics are updated self-adaptively according to the corresponding clustering group to differentiate their contributions. Experimental results on the databases of Fashion-MNIST and CIFAR10 and expression databases of FER2013 and CK+ show that the proposed clustering-based dropout achieves better accuracy than the original dropout and various dropout variants, and the most competitive performances compared with state-of-the-art algorithms.

[1]  Junmo Kim,et al.  Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[3]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[4]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[5]  Fatih Murat Porikli,et al.  Regularization of Deep Neural Networks with Spectral Dropout , 2017, Neural Networks.

[6]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[7]  Danyang Li,et al.  Ensemble of Deep Neural Networks with Probability-Based Fusion for Facial Expression Recognition , 2017, Cognitive Computation.

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Brendan J. Frey,et al.  Adaptive dropout for training deep neural networks , 2013, NIPS.

[10]  Richa Singh,et al.  Guided Dropout , 2018, AAAI.

[11]  Weihong Deng,et al.  Very deep convolutional neural network based image classification using small training sample size , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[12]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[13]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[14]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[15]  Liang Wang,et al.  Learning robust representations using recurrent neural networks for skeleton based action classification and detection , 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[16]  Xun Xu,et al.  Image Ordinal Classification and Understanding: Grid Dropout with Masking Label , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[17]  Jonathan Tompson,et al.  Efficient object localization using Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Meng Yang,et al.  Sparse deep feature learning for facial expression recognition , 2019, Pattern Recognit..

[19]  Yang Yang,et al.  ML-LocNet: Improving Object Localization with Multi-view Learning Network , 2018, ECCV.

[20]  Jane You,et al.  Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Yoshua Bengio,et al.  Challenges in Representation Learning: A Report on Three Machine Learning Contests , 2013, ICONIP.

[22]  Sida I. Wang,et al.  Dropout Training as Adaptive Regularization , 2013, NIPS.

[23]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[24]  Dae-Ki Kang,et al.  Biased Dropout and Crossmap Dropout: Learning towards effective Dropout regularization in convolutional neural network , 2018, Neural Networks.

[25]  J. Winter.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation Using the Student's t-test with extremely small sample sizes Using the Student's t-test with extremely small sample sizes , 2022 .

[26]  Yichuan Tang,et al.  Deep Learning using Support Vector Machines , 2013, ArXiv.