Reply to the Comments on Local Overfitting Control via Leverages in Jacobian Conditioning Analysis for Model Validation by I. Rivals and L. Personnaz

Jacobian Conditioning Analysis for Model Validation by Rivals and Personnaz in this issue is a comment on Monari and Dreyfus (2002). In this reply, we disprove their claims. We point to flawed reasoning in the theoretical comments and to errors and inconsistencies in the numerical examples. Our replies are substantiated by seven counterexamples, inspired by actual data, which show that the comments on the accuracy of the computation of the leverages are unsupported and that following the approach they advocate leads to discarding valid models or validating overfitted models.

[1]  Gérard Dreyfus,et al.  A machine learning approach to the estimation of the liquidus temperature of glass-forming oxide blends , 2003 .

[2]  Léon Personnaz,et al.  MLPs (Mono-Layer Polynomials and Multi-Layer Perceptrons) for Nonlinear Modeling , 2003, J. Mach. Learn. Res..

[3]  Lars Kai Hansen,et al.  Cross validation in LULOO , 1996 .

[4]  Lars Kai Hansen,et al.  Comments for: Rivals, I., & Personnaz, L. (2000). Construction of confidence intervals for neural networks based on least squares estimation. Neural Networks, 13, 463-484. , 2002, Neural networks : the official journal of the International Neural Network Society.

[5]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.

[6]  Léon Personnaz,et al.  Jacobian Conditioning Analysis for Model Validation , 2004, Neural Computation.

[7]  Gérard Dreyfus,et al.  Withdrawing an example from the training set: An analytic estimation of its effect on a non-linear parameterised model , 2000, Neurocomputing.

[8]  Léon Personnaz,et al.  CONSTRUCTION OF CONFIDENCE INTERVALS IN NEURAL MODELING USING A LINEAR TAYLOR EXPANSION , 1998 .

[9]  Lars Kai Hansen,et al.  Linear unlearning for cross-validation , 1996, Adv. Comput. Math..

[10]  Léon Personnaz,et al.  Construction of confidence intervals for neural networks based on least squares estimation , 2000, Neural Networks.

[11]  Gérard Dreyfus,et al.  Local Overfitting Control via Leverages , 2002, Neural Computation.

[12]  Douglas M. Bates,et al.  Nonlinear Regression Analysis and Its Applications , 1988 .

[13]  Gaetan Monari.  Sélection de modèles non linéaires par "leave-one-out": étude théorique et application des réseaux de neurones au procédé de soudage par points , 1999 .