A learning automata-based algorithm for determination of the number of hidden units for three-layer neural networks

There is no method to determine the optimal topology for multi-layer neural networks for a given problem. Usually the designer selects a topology for the network and then trains it. Since determination of the optimal topology of neural networks belongs to class of NP-hard problems, most of the existing algorithms for determination of the topology are approximate. These algorithms could be classified into four main groups: pruning algorithms, constructive algorithms, hybrid algorithms and evolutionary algorithms. These algorithms can produce near optimal solutions. Most of these algorithms use hill-climbing method and may be stuck at local minima. In this article, we first introduce a learning automaton and study its behaviour and then present an algorithm based on the proposed learning automaton, called survival algorithm, for determination of the number of hidden units of three layers neural networks. The survival algorithm uses learning automata as a global search method to increase the probability of obtaining the optimal topology. The algorithm considers the problem of optimization of the topology of neural networks as object partitioning rather than searching or parameter optimization as in existing algorithms. In survival algorithm, the training begins with a large network, and then by adding and deleting hidden units, a near optimal topology will be obtained. The algorithm has been tested on a number of problems and shown through simulations that networks generated are near optimal.

[1]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[2]  B. John Oommen,et al.  Graph Partitioning Using Learning Automata , 1996, IEEE Trans. Computers.

[3]  Mohammad Reza Meybodi,et al.  Call Admission Control in Cellular Mobile Networks: A Learning Automata Approach , 2002, EurAsia-ICT.

[4]  B. John Oommen,et al.  Deterministic Learning Automata Solutions to the Equipartitioning Problem , 1988, IEEE Trans. Computers.

[5]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[6]  Moshe Shoham,et al.  Approximating Functions by Neural Networks: A Constructive Solution in the Uniform Norm , 1996, Neural Networks.

[7]  Abd Rahman Ramli,et al.  Improving generalization in backpropagation networks architectures , 2005 .

[8]  J. R. Chen,et al.  Learning Algorithms: Theory and Applications in Signal Processing, Control and Communications , 2017 .

[9]  Albert Y. Zomaya,et al.  Toward generating neural network structures for function approximation , 1994, Neural Networks.

[10]  Mohammad Reza Meybodi,et al.  An adaptive call admission algorithm for cellular networks , 2005, Comput. Electr. Eng..

[11]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[12]  J. K. Kruschke,et al.  Improving generalization in backpropagation networks with distributed bottlenecks , 1989, International 1989 Joint Conference on Neural Networks.

[13]  S. Lakshmivarahan,et al.  Learning Algorithms Theory and Applications , 1981 .

[14]  Mohammad Reza Meybodi,et al.  A note on learning automata-based schemes for adaptation of BP parameters , 2002, Neurocomputing.

[15]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[16]  Dit-Yan Yeung.  Automatic determination of network size for supervised learning , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[17]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[18]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[19]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[20]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[21]  Mohammad Reza Meybodi,et al.  ADAPTIVE UNIFORM FRACTIONAL CHANNEL ALGORITHMS , 2004 .

[22]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[23]  Xiao-Hu Yu,et al.  Can backpropagation error surface not have local minima , 1992, IEEE Trans. Neural Networks.

[24]  B. John Oommen,et al.  An adaptive learning solution to the keyboard optimization problem , 1991, IEEE Trans. Syst. Man Cybern..

[25]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[26]  Kumpati S. Narendra,et al.  Adaptive and Learning Systems , 1986 .

[27]  Mohammad Reza Meybodi,et al.  Backpropagation Algorithm Adaptation Parameters Using Learning Automata , 2001, Int. J. Neural Syst..

[28]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[29]  J. M. Minor.  Parity with two layer feedforward nets , 1993, Neural Networks.

[30]  Alessandro Sperduti,et al.  Speed up learning and network optimization with extended back propagation , 1993, Neural Networks.

[31]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[32]  Shin'ichi Tamura,et al.  Capabilities of a four-layered feedforward neural network: four layers versus three , 1997, IEEE Trans. Neural Networks.

[33]  H. Beigy.  Adaptation of Momentum Factor and Steepness Parameter in Backpropagation Algorithm Using Fixed Structure Learning Automata , 2001 .

[34]  Jean-Pierre Nadal,et al.  Neural trees: a new tool for classification , 1990 .

[35]  Sebastian Thrun,et al.  The MONK''s Problems-A Performance Comparison of Different Learning Algorithms, CMU-CS-91-197, Sch , 1991 .

[36]  M. Golea,et al.  A Convergence Theorem for Sequential Learning in Two-Layer Perceptrons , 1990 .

[37]  Kumpati S. Narendra,et al.  Learning automata - an introduction , 1989 .

[38]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[39]  Mandayam A. L. Thathachar,et al.  Learning Optimal Discriminant Functions through a Cooperative Game of Automata , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[40]  Yih-Fang Huang,et al.  Bounds on the number of hidden neurons in multilayer perceptrons , 1991, IEEE Trans. Neural Networks.

[41]  Mohammad Reza Meybodi,et al.  Unsupervised learning of synaptic delays based on learning automata in an RBF-like network of spiking neurons for data clustering , 2005, Neurocomputing.

[42]  Mohammad Reza Meybodi,et al.  An Adaptive Uniform Fractional Guard Channel Algorithm: A Learning Automata Approach , 2003, IDEAL.

[43]  Mohammad Reza Meybodi,et al.  Utilization of Fixed Structure Learning Automata for Adaptation of Learning Rate in Backpropagation Algorithm , 2002 .

[44]  Yoshio Hirose,et al.  Backpropagation algorithm which varies the number of hidden units , 1989, International 1989 Joint Conference on Neural Networks.

[45]  Jeffrey Scott Vitter,et al.  Complexity Results on Learning by Neural Nets , 1991, Machine Learning.

[46]  Mohammad Reza Meybodi,et al.  SOLVING THE GRAPH ISOMORPHISM PROBLEM USING LEARNING AUTOMATA , 2000 .

[47]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[48]  Mohammad Reza Meybodi,et al.  NEW CLASSES OF LEARNING AUTOMATA BASED SCHEMES FOR ADAPTATION OF BACKPROPAGATION ALGORITHM PARAMETERS , 2001 .

[49]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[50]  Jingyuan Zhang.  Cellular Networks , 2005 .

[51]  Masahiko Arai,et al.  Bounds on the number of hidden units in binary-valued three-layer neural networks , 1993, Neural Networks.

[52]  Mohammad Reza Meybodi,et al.  A Learning Automata Based Dynamic Guard Channel Scheme , 2002, EurAsia-ICT.

[53]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[54]  Mohammad Reza Meybodi,et al.  New Learning Automata Based Algorithms for Adaptation of Backpropagation Algorithm Parameters , 2002, Int. J. Neural Syst..

[55]  P. Mars,et al.  Application of Learning Automata to Image Data Compression , 1986 .