Towards the Optimal Design of Numerical Experiments

This paper addresses the problem of the optimal design of numerical experiments for the construction of nonlinear surrogate models. We describe a new method, called learner disagreement from experiment resampling (LDR), which borrows ideas from active learning and from resampling methods: the analysis of the divergence of the predictions provided by a population of models, constructed by resampling, allows an iterative determination of the point of input space, where a numerical experiment should be performed in order to improve the accuracy of the predictor. The LDR method is illustrated on neural network models with bootstrap resampling, and on orthogonal polynomials with leave-one-out resampling. Other methods of experimental design such as random selection and D-optimal selection are investigated on the same benchmark problems.

[1]  Sebastian Thrun,et al.  Active Exploration in Dynamic Environments , 1991, NIPS.

[2]  Raymond J. Mooney,et al.  Diverse ensembles for active learning , 2004, ICML.

[3]  Greg Schohn,et al.  Less is More: Active Learning with Support Vector Machines , 2000, ICML.

[4]  D.V. Prokhorov,et al.  Multiple-start directed search for improved NN solution , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[5]  Partha Niyogi,et al.  Active Learning for Function Approximation , 1994, NIPS.

[6]  Jean-Pierre Vila,et al.  Local optimality of replications from a minimal D-optimal design in regression: A sufficient and quasi-necessary condition , 1991 .

[7]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[8]  Anthony C. Atkinson,et al.  The construction of exact D optimum wxperimental designs with application to blocking response surface designs , 1989 .

[9]  W. Näther.  Optimum experimental designs , 1994 .

[10]  Gilbert Saporta,et al.  Plans d'expériences : applications à l'entreprise , 1997 .

[11]  Marcin Witczak,et al.  Toward the training of feed-forward neural networks with the D-optimum input sequence , 2006, IEEE Transactions on Neural Networks.

[12]  Toby J. Mitchell,et al.  An Algorithm for the Construction of “D-Optimal” Experimental Designs , 2000, Technometrics.

[13]  Gérard Dreyfus,et al.  Local Overfitting Control via Leverages , 2002, Neural Computation.

[14]  Douglas M. Bates,et al.  Nonlinear Regression Analysis and Its Applications , 1988 .

[15]  Andrea Saltelli,et al.  Sensitivity analysis for model output: performance of black box techniques on three international benchmark exercises , 1992 .

[16]  Helge Toutenburg,et al.  Linear models : least squares and alternatives , 1999 .

[17]  David A. Cohn,et al.  Neural Network Exploration Using Optimal Experiment Design , 1993, NIPS.

[18]  H. Wynn.  The Sequential Generation of $D$-Optimum Experimental Designs , 1970 .

[19]  James E. Campbell,et al.  An Approach to Sensitivity Analysis of Computer Models: Part I—Introduction, Input Variable Selection and Preliminary Variable Assessment , 1981 .

[20]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[21]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[22]  J. Kiefer,et al.  Optimum Designs in Regression Problems , 1959 .

[23]  Harald Niederreiter,et al.  Random number generation and Quasi-Monte Carlo methods , 1992, CBMS-NSF regional conference series in applied mathematics.

[24]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[25]  David J. C. MacKay,et al.  Information-Based Objective Functions for Active Data Selection , 1992, Neural Computation.