Automatic tuning of hyperparameters using Bayesian optimization

Deep learning is a field in artificial intelligence that works well in computer vision, natural language processing and audio recognition. Deep neural network architectures has number of layers to conceive the features well, by itself. The hyperparameter tuning plays a major role in every dataset which has major effect in the performance of the training model. Due to the large dimensionality of data it is impossible to tune the parameters by human expertise. In this paper, we have used the CIFAR-10 Dataset and applied the Bayesian hyperparameter optimization algorithm to enhance the performance of the model. Bayesian optimization can be used for any noisy black box function for hyperparameter tuning. In this work Bayesian optimization clearly obtains optimized values for all hyperparameters which saves time and improves performance. The results also show that the error has been reduced in graphical processing unit than in CPU by 6.2% in the validation. Achieving global optimization in the trained model helps transfer learning across domains as well.

[1]  Fabio Tozeto Ramos,et al.  Bayesian optimisation for Intelligent Environmental Monitoring , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[3]  Oliver Kramer,et al.  Derivative-Free Optimization , 2011, Computational Optimization, Methods and Algorithms.

[4]  Tom Dhaene,et al.  Data-Efficient Bayesian Optimization with Constraints for Power Amplifier Design , 2018, 2018 IEEE MTT-S International Conference on Numerical Electromagnetic and Multiphysics Modeling and Optimization (NEMO).

[5]  Plamen Angelov,et al.  A generalized approach to fuzzy optimization , 1994, Int. J. Intell. Syst..

[6]  Plamen P. Angelov,et al.  An approach to automatic real‐time novelty detection, object identification, and tracking in video streams based on recursive density estimation and evolving Takagi–Sugeno fuzzy systems , 2011, Int. J. Intell. Syst..

[7]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[8]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[9]  Bernd Bischl,et al.  To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[10]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[11]  Mohammad Rouhani,et al.  Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures , 2016, ArXiv.

[12]  Nando de Freitas,et al.  On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning , 2014, AISTATS.

[13]  Plamen P. Angelov,et al.  DEC: Dynamically Evolving Clustering and Its Application to Structure Identification of Evolving Fuzzy Models , 2014, IEEE Transactions on Cybernetics.

[14]  Li Dan,et al.  Speech recognition based on convolutional neural networks , 2016, 2016 IEEE International Conference on Signal and Image Processing (ICSIP).

[15]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[16]  José Ranilla,et al.  Hyper-parameter selection in deep neural networks using parallel particle swarm optimization , 2017, GECCO.

[17]  Jan Peters,et al.  Bayesian optimization for learning gaits under uncertainty , 2015, Annals of Mathematics and Artificial Intelligence.

[18]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[19]  Dumitru Erhan,et al.  Scalable Object Detection Using Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Abdesselam Bouzerdoum,et al.  Human Motion Classification with Micro-Doppler Radar and Bayesian-Optimized Convolutional Neural Networks , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[22]  Svetha Venkatesh,et al.  Hyperparameter tuning for big data using Bayesian optimisation , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[23]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[24]  Aaron Klein,et al.  Auto-sklearn: Efficient and Robust Automated Machine Learning , 2019, Automated Machine Learning.

[25]  Xuan Zeng,et al.  An Efficient Bayesian Optimization Approach for Automated Optimization of Analog Circuits , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.