On Objective Function, Regularizer, and Prediction Error of a Learning Algorithm for Dealing With Multiplicative Weight Noise

In this paper, an objective function for training a functional link network to tolerate multiplicative weight noise is presented. Basically, the objective function is similar in form to other regularizer-based functions that consist of a mean square training error term and a regularizer term. Our study shows that under some mild conditions the derived regularizer is essentially the same as a weight decay regularizer. This explains why applying weight decay can also improve the fault-tolerant ability of a radial basis function (RBF) with multiplicative weight noise. In accordance with the objective function, a simple learning algorithm for a functional link network with multiplicative weight noise is derived. Finally, the mean prediction error of the trained network is analyzed. Simulated experiments on two artificial data sets and a real-world application are performed to verify theoretical result.

[1]  Mark J. L. Orr,et al.  Regularization in the Selection of Radial Basis Function Centers , 1995, Neural Computation.

[2]  Lionel Tarassenko,et al.  Estimations of error bounds for neural-network function approximators , 1999, IEEE Trans. Neural Networks.

[3]  Yunqian Ma,et al.  Multiple model regression estimation , 2005, IEEE Transactions on Neural Networks.

[4]  Sammy Siu,et al.  Sensitivity Analysis of the Split-Complex Valued Multilayer Perceptron Due to the Errors of the i.i.d. Inputs and Weights , 2007, IEEE Transactions on Neural Networks.

[5]  B. Liu,et al.  Error analysis of digital filters realized with floating-point arithmetic , 1969 .

[6]  Shawki Areibi,et al.  The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: A Study , 2007, IEEE Transactions on Neural Networks.

[7]  Bernard Widrow,et al.  Sensitivity of feedforward neural networks to weight errors , 1990, IEEE Trans. Neural Networks.

[8]  Toyohisa Kaneko,et al.  Effect of Coefficient Rounding in Floating-Point Digital Filters , 1971, IEEE Transactions on Aerospace and Electronic Systems.

[9]  Steve W. Piche,et al.  The selection of weight accuracies for Madalines , 1995, IEEE Trans. Neural Networks.

[10]  Andreu Català,et al.  Fault tolerance parameter model of radial basis function networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[11]  Lizhong Wu,et al.  A Smoothing Regularizer for Feedforward and Recurrent Neural Networks , 1996, Neural Computation.

[12]  Julio Ortega Lopera,et al.  Assessing the Noise Immunity and Generalization of Radial Basis Function Networks , 2004, Neural Processing Letters.

[13]  Masashi Sugiyama,et al.  Optimal design of regularization term and regularization parameter by subspace information criterion , 2002, Neural Networks.

[14]  E.E. Swartzlander,et al.  Digital neural network implementation , 1992, Eleventh Annual International Phoenix Conference on Computers and Communication [1992 Conference Proceedings].

[15]  Kang Zhang,et al.  Computation of Adalines' sensitivity to weight perturbation , 2006, IEEE Transactions on Neural Networks.

[16]  Ignacio Rojas,et al.  Improving the tolerance of multilayer perceptrons by minimizing the statistical sensitivity to weight deviations , 2000, Neurocomputing.

[17]  Jenq-Neng Hwang,et al.  Finite Precision Error Analysis of Neural Network Hardware Implementations , 1993, IEEE Trans. Computers.

[18]  Andrew Chi-Sing Leung,et al.  Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks , 2001, IEEE Trans. Neural Networks.

[19]  S. Himavathi,et al.  Feedforward Neural Network Implementation in FPGA Using Layer Multiplexing for Effective Resource Utilization , 2007, IEEE Transactions on Neural Networks.

[20]  Klaus-Robert Müller,et al.  Asymptotic statistical theory of overtraining and cross-validation , 1997, IEEE Trans. Neural Networks.

[21]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[22]  Andreu Català,et al.  Fault tolerance in the learning algorithm of radial basis function networks , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[23]  Andrew Chi-Sing Leung,et al.  On the regularization of forgetting recursive least square , 1999, IEEE Trans. Neural Networks.

[24]  Y. Takefuji,et al.  Functional-link net computing: theory, system architecture, and functionalities , 1992, Computer.

[25]  N. Bingham,et al.  Generalised Linear Models , 2010 .

[26]  Dan Simon,et al.  Distributed fault tolerance in optimal interpolative nets , 2001, IEEE Trans. Neural Networks.

[27]  Alexander J. Smola,et al.  Support Vector Method for Function Approximation, Regression Estimation and Signal Processing , 1996, NIPS.

[28]  John Moody,et al.  Note on generalization, regularization and architecture selection in nonlinear learning systems , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[29]  Jenq-Neng Hwang,et al.  Finite precision error analysis of neural network electronic hardware implementations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[30]  Chong-Ho Choi,et al.  Sensitivity analysis of multilayer perceptron with differentiable activation functions , 1992, IEEE Trans. Neural Networks.

[31]  Andrew Chi-Sing Leung,et al.  Noise-resistant fitting for spherical harmonics , 2006, IEEE Transactions on Visualization and Computer Graphics.

[32]  Chee Peng Lim,et al.  A hybrid neural network model for noisy data regression , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[33]  Salvatore Cavalieri,et al.  A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks , 1999, Neural Networks.

[34]  Kwok-Wo Wong,et al.  Generalized RLS approach to the training of neural networks , 2006, IEEE Trans. Neural Networks.

[35]  Sheng Chen,et al.  Sparse modeling using orthogonal forward regression with PRESS statistic and regularization , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[36]  Amparo Alonso-Betanzos,et al.  A measure of fault tolerance for functional networks , 2004, Neurocomputing.

[37]  Yoh-Han Pao,et al.  The functional link net and learning optimal control , 1995, Neurocomputing.

[38]  Nigel Steele,et al.  On parity problems and the functional-link artificial neural network , 2005, Neural Computing & Applications.

[39]  Sheng Chen,et al.  Local regularization assisted orthogonal least squares regression , 2006, Neurocomputing.

[40]  ImplementationsJames B. BurrDepartment.  Digital Neural Network Implementations , 1995 .

[41]  Ignacio Rojas,et al.  Obtaining Fault Tolerant Multilayer Perceptrons Using an Explicit Regularization , 2000, Neural Processing Letters.

[42]  Ignacio Rojas,et al.  An Accurate Measure for Multilayer Perceptron Tolerance to Weight Deviations , 1999, Neural Processing Letters.

[43]  Shun-ichi Amari,et al.  Network information criterion-determining the number of hidden units for an artificial neural network model , 1994, IEEE Trans. Neural Networks.

[44]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[45]  Sameer Singh.  Noise impact on time-series forecasting using an intelligent pattern matching technique , 1999, Pattern Recognit..

[46]  Ignacio Rojas,et al.  A Quantitative Study of Fault Tolerance, Noise Immunity, and Generalization Ability of MLPs , 2000, Neural Computation.

[47]  H. Akaike.  A new look at the statistical model identification , 1974 .