Toward constructive methods for sigmoidal neural networks - function approximation in engineering mechanics applications

This paper reports a continuous development of the work by the authors presented at IJCNN 2005 & 2007 [1, 2]. A series of parsimonious universal approximator architectures with pre-defined values for weights and biases called “neural network prototypes” are proposed and used in a repetitive and systematic manner for the initialization of sigmoidal neural networks in function approximation. This paper provides a more in-depth literature review, presents one training example using laboratory data indicating quick convergence and trained sigmoidal neural networks with stable generalization capability, and discusses the complexity measure in [3, 4]. This study centers on approximating a subset of static nonlinear target functions - mechanical restoring force considered as a function of system states (displacement and velocity) for single-degree-of-freedom systems. We strive for efficient and rigorous constructive methods for sigmoidal neural networks to solve function approximation problems in this engineering mechanics application and beyond. Future work is identified.

[1]  Pascal Larzabal,et al.  Initialization of Supervised Training for Parametric Estimation , 1999, Neural Processing Letters.

[2]  Zoran Gajic,et al.  Linear Dynamic Systems and Signals , 2002 .

[3]  Malcolm James Beynon,et al.  Neural networks and flexible approximations , 2000 .

[4]  Ah Chung Tsoi,et al.  Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results , 1998, Neural Networks.

[5]  Jin-Song Pei,et al.  Constructing Multilayer Feedforward Neural Networks to Approximate Nonlinear Functions in Engineering Mechanics Applications , 2008 .

[6]  J. Schoukens,et al.  Parametric and nonparametric identification of linear systems in the presence of nonlinear distortions-a frequency domain approach , 1998, IEEE Trans. Autom. Control..

[7]  Sami F. Masri,et al.  Development of data-based model-free representation of non-conservative dissipative systems , 2007 .

[8]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[9]  Robert M. Farber,et al.  How Neural Nets Work , 1987, NIPS.

[10]  J.M.A. Scherpen,et al.  Multidomain modeling of nonlinear networks and systems , 2009, IEEE Control Systems.

[11]  Andrew W. Smyth,et al.  Mapping polynomial fitting into feedforward neural networks for modeling nonlinear dynamic systems and beyond , 2005 .

[12]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[13]  Niels Kjølstad Poulsen,et al.  Neural Networks for Modelling and Control of Dynamic Systems: A Practitioner’s Handbook , 2000 .

[14]  A. Barron.  Approximation and Estimation Bounds for Artificial Neural Networks , 1991, COLT '91.

[15]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[16]  S. Osowski,et al.  New approach to selection of initial values of weights in neural function approximation , 1993 .

[17]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[18]  Paul J. Werbos,et al.  The roots of backpropagation , 1994 .

[19]  Donald A. Sofge,et al.  Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches , 1992 .

[20]  Andrew W. Smyth,et al.  New Approach to Designing Multilayer Feedforward Neural Network Architecture for Modeling Nonlinear Restoring Forces. I: Formulation , 2006 .

[21]  T. T. Baber,et al.  Modeling General Hysteresis Behavior and Random Vibration Application , 1986 .

[22]  Steven A. Orszag,et al.  CBMS-NSF REGIONAL CONFERENCE SERIES IN APPLIED MATHEMATICS , 1978 .

[23]  L. K. Jones,et al.  The computational intractability of training sigmoidal neural networks , 1997, IEEE Trans. Inf. Theory.

[24]  Tommy W. S. Chow,et al.  Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients , 2001, IEEE Trans. Neural Networks.

[25]  R. Bouc.  Forced Vibration of Mechanical Systems with Hysteresis , 1967 .

[26]  Jin-Song Pei,et al.  Mapping some functions and four arithmetic operations to multilayer feedforward neural networks , 2008, SPIE Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[27]  Andrew W. Smyth,et al.  Neural Network Initialization with Prototypes - Function Approximation in Engineering Mechanics Applications , 2007, 2007 International Joint Conference on Neural Networks.

[28]  L. Jones.  Constructive approximations for neural networks by sigmoidal functions , 1990, Proc. IEEE.

[29]  Thierry Denoeux,et al.  Initializing back propagation networks with prototypes , 1993, Neural Networks.

[30]  Mikko Lehtokangas,et al.  Fast initialization for cascade-correlation learning , 1999, IEEE Trans. Neural Networks.

[31]  Leonardo Franco,et al.  Constructive Neural Networks , 2009, Constructive Neural Networks.

[32]  Chuanyi Ji,et al.  Performance and efficiency: recent advances in supervised learning , 1999, Proc. IEEE.

[33]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[34]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[35]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[36]  Y. Wen.  Method for Random Vibration of Hysteretic Systems , 1976 .

[37]  Jin-Song Pei,et al.  Neural network initialization with prototypes - a case study in function approximation , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[38]  Mohammad Noori,et al.  Random Vibration of Degrading, Pinching Systems , 1985 .

[39]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[40]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[41]  Robert F. Stengel,et al.  Smooth function approximation using neural networks , 2005, IEEE Transactions on Neural Networks.

[42]  Sami F. Masri,et al.  A Nonparametric Identification Technique for Nonlinear Dynamic Problems , 1979 .