Ensemble of Surrogates: a Framework based on Minimization of the Mean Integrated Square Error

In engineering problems, high-fidelity simulations are often time consuming and computationally expensive. Surrogate models, frequently chosen based on past experience, are commonly used to replace expensive simulations. However, when a single surrogate is needed, previous work has shown that it is a good strategy to fit a large set of different surrogates and pick one based on cross-validation errors (PRESS in particular). In addition, cross validation errors may also be used to create a weighted surrogate. In this paper, we investigate how well PRESS estimates the RMS errors, and whether to use the best PRESS solution or a weighted surrogate when a single surrogate is needed. We propose the minimization of the integrated square error as a way to compute the weights of the weighted average surrogate. The investigation is based on a set of standard test functions as well as an industrial example of the lift, drag and pitching moment coefficients of an airfoil. We confirm that it pays to generate a large set of different surrogates and then use PRESS as a criterion for selection. We find that with sufficient number of points (i) cross validation error vectors provide an excellent estimate of the RMS errors, and (ii) in general, PRESS is good for filtering out inaccurate surrogates, and especially in high dimensions, PRESS is also good for identifying the best surrogate of the set. However, it appears that the potential gains from using weighted surrogates diminish substantially in high dimensions. Surprisingly, this result holds even if we used perfect knowledge of the function to select the best possible weights. Additionally, we also found that PRESS as obtained through the k-fold strategy successfully estimates the RMS errors.