Comparison of Various Learning Rate Scheduling Techniques on Convolutional Neural Network

The learning rate is a hyperparameter which determines how much the model should change concerning the error each time the model parameters are updated. It is important to tune the learning rate properly because if it is set too low, our model will converge very slowly and if set too high, our model may diverge from the optimal error point. Some conventional learning rate tuning techniques include constant learning rate, step decay, cyclical learning rate and many more. In this paper, we have implemented some of these techniques and compared the model performances gained using these techniques.

[1]  Qionghai Dai,et al.  Exponential decay sine wave learning rate for fast deep neural network training , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[2]  S. Roy,et al.  Factors influencing the choice of a learning rate for a backpropagation neural network , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[3]  Jiwen Dong,et al.  Simple convolutional neural network on image classification , 2017, 2017 IEEE 2nd International Conference on Big Data Analysis (ICBDA)(.

[4]  Lucas Lima,et al.  Efficient Hyperparameter Optimization of Convolutional Neural Networks on Classification of Early Pulmonary Nodules , 2019, 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS).

[5]  M. Yoshida,et al.  Tuning of learning rate and momentum on backpropagation , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[6]  Purnendu Mishra,et al.  Polynomial Learning Rate Policy with Warm Restart for Deep Neural Network , 2019, TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON).

[7]  Shu-Hung Leung,et al.  Convergence of the generalized back-propagation algorithm with constant learning rates , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[8]  Leslie N. Smith,et al.  Cyclical Learning Rates for Training Neural Networks , 2015, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[9]  Siddhant Khandelwal,et al.  Analysis of Hyperparameter Tuning in Neural Style Transfer , 2018, 2018 Fifth International Conference on Parallel, Distributed and Grid Computing (PDGC).

[10]  Thomas Villmann,et al.  Precision-Recall-Optimization in Learning Vector Quantization Classifiers for Improved Medical Classification Systems , 2014, 2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).