Simulation selection for empirical model comparison

We propose an efficient statistical method for the empirical model comparison, which is typically referred to as a simulation procedure to evaluate multiple statistical learning algorithms. First, we use experimental designs to appropriately construct the training and test sets for estimating the empirical performances of these models using the mean square errors. Second, we apply the idea of Bayesian fully sequential ranking and selection to optimally allocate the simulation budget according to the value of information. To make the procedure computationally tractable, we assume a normal-Wishart prior distribution, and propose a new approximation scheme for the posterior distribution by matching it with a normal-Wishart distribution using the first-order moment. Numerical experiments are conducted to show the superiority of the proposed approach on empirical model comparison problems.