Developing Practical Neural Network Applications Using Back‐Propagation

: In the past few years, neural networks have emerged as a problem-solving technique with capabilities suited to many civil engineering problems. Among the various neural network paradigms available, back-propagation is by far the most utilized for its relatively simple mathematical proofs and good generalization capabilities. Despite its capabilities, back-propagation suffers from several problems that hinder the development of practical neural network applications. These include slow training, ill-defined knowledge representation and problem structuring, and nonguided design of an optimal network configuration for adequate generalization. This paper represents an effort to guide the process of developing practical neural network applications using back-propagation. The paper starts with a brief description of back-propagation mathematics. Some of the heuristics and techniques used to overcome back-propagation problems, particularly lack of generalization, are identified and outlined, along with areas of potential improvements to the paradigm. An application development methodology is proposed utilizing the identified heuristics and techniques. The methodology provides a structured framework for designing and implementing practical neural network applications with less effort.

[1]  Jim Howell.  Inside a neural network , 1990 .

[2]  David E. Goldberg,et al.  Genetic Algorithms in Pipeline Optimization , 1987 .

[3]  Thomas P. Vogl,et al.  Rescaling of variables in back propagation learning , 1991, Neural Networks.

[4]  John Shawe-Taylor,et al.  Sample sizes for multiple-output threshold networks , 1991 .

[5]  Maureen Caudill,et al.  Neural networks primer, part III , 1988 .

[6]  Maureen Caudill.  Expert networks , 1990 .

[7]  D Zipser,et al.  Learning the hidden structure of speech. , 1988, The Journal of the Acoustical Society of America.

[8]  R. D. Vanluchene,et al.  Neural Networks in Structural Engineering , 1990 .

[9]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[10]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[11]  Maureen Caudill,et al.  Evolutionary neural networks , 1991 .

[12]  David Bailey,et al.  How to develop neural-network applications , 1990 .

[13]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[14]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[15]  Tarek Hegazy,et al.  Markup estimation using neural network methodology , 1993 .

[16]  Maureen Caudill,et al.  Neural network training tips and techniques , 1991 .

[17]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[18]  Frank Fallside,et al.  An adaptive training algorithm for back propagation networks , 1987 .

[19]  Tarek Hegazy,et al.  Potential applications of neural networks in construction , 1992 .

[20]  Halbert White,et al.  Neural-network learning and statistics , 1989 .

[21]  Richard P. Brent,et al.  Fast training algorithms for multilayer neural nets , 1991, IEEE Trans. Neural Networks.

[22]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[23]  Don Tveter.  Better speed through integers , 1990 .

[24]  David Bailey,et al.  Developing neural-network applications , 1990 .

[25]  Tarek Hegazy,et al.  Neural networks as tools in construction , 1991 .

[26]  Michael K. Weir,et al.  A method for self-determination of adaptive learning rates in back propagation , 1991, Neural Networks.

[27]  Leslie G. Valiant,et al.  A general lower bound on the number of examples needed for learning , 1988, COLT '88.

[28]  P. Lisboa,et al.  Complete solution of the local minima in the XOR problem , 1991 .

[29]  Yoshio Hirose,et al.  Backpropagation algorithm which varies the number of hidden units , 1989, International 1989 Joint Conference on Neural Networks.

[30]  Derek F. Stubbs.  Three applications of neurocomputing in biomedical research , 1989, International 1989 Joint Conference on Neural Networks.

[31]  Scott Austin,et al.  An introduction to genetic algorithms , 1990 .

[32]  G. David Garson,et al.  Interpreting neural-network connection weights , 1991 .