On Predicting the Optimal Number of Hidden Nodes

Determining the optimal number of hidden nodes is the most challenging aspect of Artificial Neural Network (ANN) design. To date, there are still no reliable methods of determining this a priori, as it depends on so many domain-specific factors. Current methods which take these into account, such as exhaustive search, growing and pruning and evolutionary algorithms are not only inexact, but also extremely time consuming -- in some cases prohibitively so. A novel approach embodied in a system called Heurix is introduced. This rapidly predicts the optimal number of hidden nodes from a small number of sample topologies. It can be configured to favour speed (low complexity), accuracy, or a balance between the two. Single hidden layer feedforward networks (SLFNs) can be built twenty times faster, and with a generalisation error of as little as 0.4% greater than those found by exhaustive search.

[1]  V. Tikhomirov.  On the Representation of Continuous Functions of Several Variables as Superpositions of Continuous Functions of one Variable and Addition , 1991 .

[2]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[3]  L. Goddard.  Approximation of Functions , 1965, Nature.

[4]  Adam Blum,et al.  Neural Networks in C++: An Object-Oriented Framework for Building Connectionist Systems , 1992 .

[5]  Katsunari Shibata,et al.  Effect of number of hidden neurons on learning in large-scale layered neural networks , 2009, 2009 ICCAS-SICE.

[6]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[7]  G. Lorentz.  Approximation of Functions , 1966 .

[8]  Chao He,et al.  Notice of Retraction Emission modeling of diesel engine fueled with biodiesel based on back propagation neural network , 2010 .

[9]  Jinchuan Ke,et al.  Empirical Analysis of Optimal Hidden Neurons in Neural Network Modeling for Stock Prediction , 2008, 2008 IEEE Pacific-Asia Workshop on Computational Intelligence and Industrial Application.

[10]  Peter L. Bartlett,et al.  For Valid Generalization the Size of the Weights is More Important than the Size of the Network , 1996, NIPS.

[11]  D. Sprecher.  On the structure of continuous functions of several variables , 1965 .

[12]  Shin'ichi Tamura,et al.  Capabilities of a four-layered feedforward neural network: four layers versus three , 1997, IEEE Trans. Neural Networks.

[13]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[14]  Vera Kurková,et al.  Kolmogorov's theorem and multilayer neural networks , 1992, Neural Networks.

[15]  Ying Zhang,et al.  Fast unit pruning algorithm for feedforward neural network design , 2008, Appl. Math. Comput..

[16]  Kevin Swingler,et al.  Applying neural networks - a practical guide , 1996 .

[17]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[18]  Michael J. A. Berry,et al.  Data mining techniques - for marketing, sales, and customer support , 1997, Wiley computer publishing.

[19]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.