Estimating Optimal Depth of VGG Net with Tree-Structured Parzen Estimators

Deep convolutional neural networks (CNNs) have shown astonishing performances in variety of fields. However, different architectures of the networks are required for different datasets, and finding right architecture for given data has been a topic of great interest in computer vision communities. One of the most important factors of the CNNs architecture is the depth of the networks, which plays a significant role in avoiding over-fitting. Grid Search is widely used for estimating the depth, but it requires huge computation time. Motivated by this, a method for finding an optimal architecture depth is introduced, which is based on a hyper-parameter optimizer called Tree-Structured Parzen Estimators (TPE). In this work, we show that the TPE is capable of estimating the CNNs architecture depth with an accuracy of 83.33% with CIFAR-10 dataset and 60.00% with CIFAR-100 dataset while it reduces the computation time by more 70% compared to the Grid Search.

[1]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[4]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[5]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[6]  Yann LeCun,et al.  Very Deep Convolutional Networks for Text Classification , 2016, EACL.

[7]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[8]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[9]  Dianhui Wang,et al.  Randomness in neural networks: an overview , 2017, WIREs Data Mining Knowl. Discov..

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.