Two Hidden Layers are Usually Better than One

This study investigates whether feedforward neural networks with two hidden layers generalise better than those with one. In contrast to the existing literature, a method is proposed which allows these networks to be compared empirically on a hidden-node-by-hidden-node basis. This is applied to ten public domain function approximation datasets. Networks with two hidden layers were found to be better generalisers in nine of the ten cases, although the actual degree of improvement is case dependent. The proposed method can be used to rapidly determine whether it is worth considering two hidden layers for a given problem.

[1]  Kurt Hornik,et al.  Some new results on neural network approximation , 1993, Neural Networks.

[2]  Miltos Petridis,et al.  Accelerated Optimal Topology Search for Two-Hidden-Layer Feedforward Neural Networks , 2016, EANN.

[3]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[4]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[5]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[6]  Takéhiko Nakama,et al.  Comparisons of Single- and Multiple-Hidden-Layer Neural Networks , 2011, ISNN.

[7]  I-Cheng Yeh,et al.  Modeling of strength of high-performance concrete using artificial neural networks , 1998 .

[8]  Claire Mathieu,et al.  Multilayer Neural Networks: One or Two Hidden Layers? , 1996, NIPS.

[9]  Miltiadis Petridis,et al.  On the Optimal Node Ratio between Hidden Layers: A Probabilistic Study , 2016 .

[10]  Guoqiang Peter Zhang,et al.  Avoiding Pitfalls in Neural Network Research , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Mark Beale,et al.  Neural Network Toolbox™ User's Guide , 2015 .

[12]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[13]  Eduardo D. Sontag,et al.  Feedback Stabilization Using Two-Hidden-Layer Nets , 1991, 1991 American Control Conference.