Comparison of Regularization Methods for ImageNet Classification with Deep Convolutional Neural Networks

Large and Deep Convolutional Neural Networks achieve good results in image classification tasks, but they need methods to prevent overfitting. In this paper we compare performance of different regularization techniques on ImageNet Large Scale Visual Recognition Challenge 2013. We show empirically that Dropout works better than DropConnect on ImageNet dataset.

[1]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[2]  Yichuan Tang,et al.  Deep Learning using Linear Support Vector Machines , 2013, 1306.0239.

[3]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[4]  Rob Fergus,et al.  Stochastic Pooling for Regularization of Deep Convolutional Neural Networks , 2013, ICLR.

[5]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[6]  Joseph JáJá,et al.  From Maxout to Channel-Out: Encoding Information on Sparse Pathways , 2013, ICANN.

[7]  Dumitru Erhan,et al.  Deep Neural Networks for Object Detection , 2013, NIPS.

[8]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Neural Networks , 2013 .

[9]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[10]  Razvan Pascanu,et al.  Learned-norm pooling for deep neural networks , 2013, ArXiv.

[11]  Tara N. Sainath,et al.  Improvements to Deep Convolutional Neural Networks for LVCSR , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[12]  Marc'Aurelio Ranzato,et al.  DeViSE: A Deep Visual-Semantic Embedding Model , 2013, NIPS.

[13]  Brendan J. Frey,et al.  Adaptive dropout for training deep neural networks , 2013, NIPS.

[14]  Jakub M. Tomczak.  Prediction of breast cancer recurrence using Classification Restricted Boltzmann Machine with Dropping , 2013, ArXiv.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.