A practical overview of neural networks

This paper overviews the myths and misconceptions that have surrounded neural networks in recent years. Focusing on backpropagation and the Hopfield network, we discuss the problems that have plagued practical application of these techniques, and review some of the recent progress made. Both real and perceived inadequacies of backpropagation are discussed, as well as the need for an understanding of statistics and of the problem domain in order to apply and assess the neural network properly. We consider alternatives or variants to backpropagation, which overcome some of its real limitations. The Hopfield network's poor performance on the traveling salesman problem in combinatorial optimization has colored its reception by engineers; we describe both new research in this area and promising results in other practical optimization applications. Overall, it is hoped, this paper will aid in a more balanced understanding of neural networks. They seem worthy of consideration in many applications, but they do not deserve the status of a panacea – nor are they as fraught with problems as would now seem to be implied.

[1]  Cihan H. Dagli,et al.  Neural Networks in Practice: Survey Results , 1997 .

[2]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[3]  Laurene V. Fausett,et al.  Fundamentals Of Neural Networks , 1993 .

[4]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[5]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[6]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[7]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[8]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[9]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[10]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[11]  Vasant G Honavar,et al.  MTiling A Constructive Neural Network Learning Algorithm for Multi Category Pattern Classi cation , 1996 .

[12]  Marimuthu Palaniswami,et al.  A hybrid neural approach to combinatorial optimization , 1996, Comput. Oper. Res..

[13]  S. S. Sengupta,et al.  The traveling salesman problem , 1961 .

[14]  Laura I. Burke,et al.  Tool condition monitoring in metal cutting: A neural network approach , 1991, J. Intell. Manuf..

[15]  John E. Moody,et al.  The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems , 1991, NIPS.

[16]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[17]  Aart Joppe,et al.  A neural network for solving the travelling salesman problem on the basis of city adjacency in the tour , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[18]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[19]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Pattern Classification , 1995 .