Classification Performance Analysis of Weight Update Method Applied to Various ConvNet Models

Research on the artificial intelligence is increasing with the improvement of computing power and the development of algorithm theory. In particular, the deep neural network, which is a field of machine learning, is widely used in artificial intelligence because it can process data that cannot be solved by conventional shallow neural networks more effectively. Implementation of a deep neural network is generally based on popularized neural networks with excellent generalization performance, which saves time and effort. However, it is difficult to guess which deep neural networks and optimization methods can achieve the best performance in their dataset. In this paper, we have tested the four convolutional neural networks (ConvNet) and four weight update methods. Experiments were conducted using a 5-fold cross-validation based on insect image dataset. As a result, the ResNet-50 and AdaDelta combination showed the best performance (89.98 ± 1.40)% in the insect dataset.

[1]  Léon Bottou,et al.  Stochastic Gradient Descent Tricks , 2012, Neural Networks: Tricks of the Trade.

[2]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[3]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[4]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[7]  Robert Hecht-Nielsen,et al.  Theory of the backpropagation neural network , 1989, International 1989 Joint Conference on Neural Networks.

[8]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[9]  Han Yu,et al.  New applications of image classification in character recognition , 2017, 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE).

[10]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[12]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[13]  Pariwat Ongsulee,et al.  Artificial intelligence, machine learning and deep learning , 2017, 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE).

[14]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[17]  A. Roli.  Artificial Neural Networks , 2012, Lecture Notes in Computer Science.

[18]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.