MODE-CNN: A fast converging multi-objective optimization algorithm for CNN-based models

Abstract Convolutional neural networks (CNNs) have been used to solve many problems in computer science with a high level of success, and have been applied in many fields in recent years. However, most of the designs of these models are still tuned manually; obtaining the highest performing CNN model is therefore very time-consuming, and is sometimes not achievable. Recently, researchers have started using optimization algorithms for the automatic adjustment of the hyper-parameters of CNNs. In particular, single-objective optimization algorithms have been used to achieve the highest network accuracy for the design of a CNN. When these studies are examined, it can be seen that the most significant problem in the optimization of the parameters of a CNN is that a great deal of time is required for tuning. Hence, optimization algorithms with high convergence rates are needed for the parameter optimization of deep networks. In this study, we first develop an algorithm called MODE-CNN, based on the multi-objective differential evolution (MODE) algorithm for parameter optimization of CNN or CNN-based methods. MODE-CNN is then compared with four different multi-objective optimization algorithms. This comparison is carried out using 16 benchmark functions and four different metrics, with 100 independent runs. It is observed that the algorithm is robust and competitive compared to alternative approaches, in terms of its accuracy and convergence. Secondly, the MODE-CNN algorithm is used in the parameter optimization of a CNN-based method, developed previously by the authors, for the segmentation and classification of medical images. In this method, there are three parameters that influence the test time and accuracy: the general stride (GS), neighbour distance (ND), and patch accuracy (PA). These parameters need to be optimized to give the highest possible accuracy and lowest possible test time. With the MODE-CNN algorithm, the most appropriate GS, ND, and PA values are obtained for the test time and accuracy. As a result, it is observed that the MODE-CNN algorithm is successful, both in comparison with multi-objective algorithms and in the parameter optimization of a CNN-based method.

[1]  Andrzej Osyczka,et al.  7 – Multicriteria optimization for engineering design , 1985 .

[2]  João Paulo Papa,et al.  A multi-objective artificial butterfly optimization approach for feature selection , 2020, Appl. Soft Comput..

[3]  Masoud Daneshtalab,et al.  DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems , 2020, Microprocess. Microsystems.

[4]  Saúl Zapotecas Martínez,et al.  Multi-objective grey wolf optimizer based on decomposition , 2019, Expert Syst. Appl..

[5]  Elliot Meyerson,et al.  Evolving Deep Neural Networks , 2017, Artificial Intelligence in the Age of Neural Networks and Brain Computing.

[6]  Enrique Alba,et al.  SMPSO: A new PSO-based metaheuristic for multi-objective optimization , 2009, 2009 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making(MCDM).

[7]  José Ranilla,et al.  Particle swarm optimization for hyper-parameter selection in deep neural networks , 2017, GECCO.

[8]  Gary G. Yen,et al.  Particle swarm optimization of deep neural networks architectures for image classification , 2019, Swarm Evol. Comput..

[9]  Jay Prakash,et al.  NSABC: Non-dominated sorting based multi-objective artificial bee colony algorithm and its application in data clustering , 2016, Neurocomputing.

[10]  Diego Oliva,et al.  A new multi-objective optimization algorithm combined with opposition-based learning , 2021, Expert Syst. Appl..

[11]  W. Du,et al.  Multi-objective differential evolution with ranking-based mutation operator and its application in chemical process optimization , 2014 .

[12]  Vimal J. Savsani,et al.  Non-dominated sorting moth flame optimization (NS-MFO) for multi-objective problems , 2017, Eng. Appl. Artif. Intell..

[13]  Wei Wei,et al.  2019 Formatting Instructions for Authors Using LaTeX , 2018 .

[14]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[15]  Kalyanmoy Deb,et al.  Self-Adaptive Genetic Algorithms with Simulated Binary Crossover , 2001, Evolutionary Computation.

[16]  Marco Laumanns,et al.  Scalable Test Problems for Evolutionary Multiobjective Optimization , 2005, Evolutionary Multiobjective Optimization.

[17]  Peter J. Fleming,et al.  Multiobjective optimization and multiple constraint handling with evolutionary algorithms. I. A unified formulation , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[18]  Halife Kodaz,et al.  A new algorithm based on gray wolf optimizer and shuffled frog leaping algorithm to solve the multi-objective optimization problems , 2020, Appl. Soft Comput..

[19]  H. Abbass,et al.  PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[20]  Frank Kursawe,et al.  A Variant of Evolution Strategies for Vector Optimization , 1990, PPSN.

[21]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Darrell Whitley,et al.  A genetic algorithm tutorial , 1994, Statistics and Computing.

[23]  Carlos A. Coello Coello,et al.  An Algorithm Based on Differential Evolution for Multi-Objective Problems , 2005 .

[24]  Masanori Suganuma,et al.  A genetic programming approach to designing convolutional neural network architectures , 2017, GECCO.

[25]  Carlos A. Coello Coello,et al.  Improving PSO-Based Multi-objective Optimization Using Crowding, Mutation and epsilon-Dominance , 2005, EMO.

[26]  Martin J. Oates,et al.  PESA-II: region-based selection in evolutionary multiobjective optimization , 2001 .

[27]  Mustafa Servet Kiran,et al.  A multi-objective artificial algae algorithm , 2018, Appl. Soft Comput..

[28]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[29]  Kalyanmoy Deb,et al.  NSGA-Net: neural architecture search using multi-objective genetic algorithm , 2018, GECCO.

[30]  Mengjie Zhang,et al.  Evolving Deep Convolutional Neural Networks for Image Classification , 2017, IEEE Transactions on Evolutionary Computation.

[31]  Leandro dos Santos Coelho,et al.  Multi-objective grey wolf optimizer: A novel algorithm for multi-criterion optimization , 2016, Expert Syst. Appl..

[32]  Arthur C. Sanderson,et al.  Pareto-based multi-objective differential evolution , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[33]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[34]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[35]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[36]  Quoc V. Le,et al.  Large-Scale Evolution of Image Classifiers , 2017, ICML.

[37]  Oriol Vinyals,et al.  Hierarchical Representations for Efficient Architecture Search , 2017, ICLR.

[38]  Benjamín Barán,et al.  Performance metrics in multi-objective optimization , 2015, 2015 Latin American Computing Conference (CLEI).

[39]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[40]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[41]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[42]  D. Goldberg,et al.  BOA: the Bayesian optimization algorithm , 1999 .

[43]  Adam Lipowski,et al.  Roulette-wheel selection via stochastic acceptance , 2011, ArXiv.

[44]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[45]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[46]  Antonio J. Nebro,et al.  jMetal: A Java framework for multi-objective optimization , 2011, Adv. Eng. Softw..

[47]  Bahriye Akay,et al.  Synchronous and asynchronous Pareto-based multi-objective Artificial Bee Colony algorithms , 2012, Journal of Global Optimization.

[48]  Ahmet zk,et al.  A novel metaheuristic for multi-objective optimization problems , 2017 .

[49]  Alan L. Yuille,et al.  Genetic CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[50]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[51]  Martin J. Oates,et al.  The Pareto Envelope-Based Selection Algorithm for Multi-objective Optimisation , 2000, PPSN.

[52]  Erkan Ülker,et al.  A new method for automatic counting of ovarian follicles on whole slide histological images based on convolutional neural network , 2019, Comput. Biol. Medicine.

[53]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Carlos A. Coello Coello,et al.  Multi-Objective Particle Swarm Optimizers: An Experimental Comparison , 2009, EMO.