Aspects of network training and validation on noisy data - Part 1. Training aspects

Abstract Multi-layered feed-forward (MLF) neural networks are commonly trained by the generalized Delta learning rule. The training method has shown to be robust and easy to implement for various chemical problems in analytical chemistry. However, the slow and unpredictable learning behavior puts considerable limitations to the interpretation and validation of neural network models. In this two-part paper, both training and validation aspects are addressed. The first part of this paper focuses to the generalized Delta learning rule and some important aspects of network training. It is shown that the use of quasi-Newton training on unfolded MLF networks, accelerates the training time to an order of magnitude. The learning behavior of MLF-networks trained by the generalised Delta rule and the quasi-Newton learning rule are compared by means of simulated and real data from analytical chemistry. Some conclusions about the general applicability of the discussed methods are drawn.

[1]  L.M.C. Buydens,et al.  Aspects of network training and validation on noisy data: Part 2. Validation aspects , 1998 .

[2]  Seymour Geisser,et al.  The Predictive Sample Reuse Method with Applications , 1975 .

[3]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[4]  D. M. Titterington,et al.  Neural Networks: A Review from a Statistical Perspective , 1994 .

[5]  Peter L. Bonate,et al.  Approximate confidence intervals in calibration using the bootstrap , 1993 .

[6]  Roberto Todeschini,et al.  The data analysis handbook , 1994, Data handling in science and technology.

[7]  Martin T. Hagan,et al.  Neural network design , 1995 .

[8]  H. Lohninger.  Neural Networks for Chemists. Von J. Zupan und J. Gasteiger. VCH Verlagsgesellschaft mbH 1993. 305 S., mit zahlr. Abb. und Tab., geb., DM 68,–. , 1994 .

[9]  Gerrit Kateman,et al.  Neural networks used as a soft-modelling technique for quantitative description of the relation between physical structure and mechanical properties of poly(ethylene terephthalate) yarns , 1992 .

[10]  Timothy Masters,et al.  Advanced algorithms for neural networks: a C++ sourcebook , 1995 .

[11]  Luke E. K. Achenie,et al.  On the Use of Quasi-Newton-Based Training of a Feedforward Neural Network for Time Series Forecasting , 1995, J. Intell. Fuzzy Syst..

[12]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[13]  E.A.H. Timmermans,et al.  Adaptive noise cancellation on inductively coupled plasma spectroscopy , 1997 .

[14]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[15]  S. Wold.  Cross-Validatory Estimation of the Number of Components in Factor and Principal Components Models , 1978 .

[16]  L.M.C. Buydens,et al.  Response to "Comment on a recent sensitivity analysis of radial base function and multi-layer feed-forward neural network models" , 1996 .

[17]  S. Buckland,et al.  Monte Carlo Methods For Confidence Interval Estimation Using The Bootstrap Technique , 1983 .

[18]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[19]  Gerrit Kateman,et al.  Using artificial neural networks for solving chemical problems Part I. Multi-layer feed-forward networks , 1994 .

[20]  N. Draper,et al.  Applied Regression Analysis , 1966 .

[21]  Monica Th. Markus,et al.  Bootstrap confidence regions in nonlinear multivariate analysis , 1994 .

[22]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[23]  Edwin Metcalfe,et al.  Optimisation of radial basis and backpropagation neural networks for modelling auto-ignition temperature by quantitative-structure property relationships , 1996 .

[24]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[25]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[26]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[27]  Barry J. Wythoff,et al.  Backpropagation neural networks , 1993 .

[28]  Lutgarde M. C. Buydens,et al.  Parallel Processing of Chemical Information in a Local Area Network - II. a Parallel Cross-validation Procedure for Artificial Neural Networks , 1996, Comput. Chem..

[29]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.