Analysis and pruning of nonlinear auto-association networks

In the paper, an analysis of a three‐layer nonlinear auto‐association network with linear output neurons and sigmoidal hidden neurons is carried out. Simulations have shown that the hidden layer neurons of this network operate mainly in their linear region. By studying the statistical relations governing the operation of such a network, the nearly linear behaviour of the sigmoidal hidden neurons was verified. Dealing with the network as being totally linear, a pruning algorithm is proposed to find out the minimum number of hidden neurons needed to reconstruct the input data within a certain error threshold. The performance of the pruning algorithm is illustrated with two examples.

[1]  H. Saunders,et al.  Probability, Random Variables and Stochastic Processes (2nd Edition) , 1989 .

[2]  H. Bourlard,et al.  Auto-association by multilayer perceptrons and singular value decomposition , 1988, Biological Cybernetics.

[3]  Garrison W. Cottrell,et al.  Image compression by back-propagation: An example of extensional programming , 1988 .

[4]  Robert Azencott,et al.  Image compression with back propagation: Improvement of the visual restoration using different cost functions , 1991, Neural Networks.

[5]  F. R. Gantmakher.  The Theory of Matrices , 1984 .

[6]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[7]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[8]  Bayya Yegnanarayana,et al.  Analysis of autoassociative mapping neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[9]  Leonid Kruglyak,et al.  How to Solve the N Bit Encoder Problem with Just Two Hidden Units , 1990, Neural Computation.

[10]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[11]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[12]  Athanasios Papoulis,et al.  Probability, Random Variables and Stochastic Processes , 1965 .

[13]  Nathalie Japkowicz,et al.  Nonlinear Autoassociation Is Not Equivalent to PCA , 2000, Neural Computation.

[14]  Emile Fiesler.  Minimal and High Order Neural Network Topologies , 1993 .

[15]  Pascal Blanchet.  Data Compression Using Multilayer Perceptrons , 1989, NATO Neurocomputing.

[16]  C. Lee Giles,et al.  Neural Information Processing Systems 7 , 1995 .

[17]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.