Information metrics for model selection in function estimation

A model selection framework is presented for function estimation under limited information, where only a small set of (noisy) data points are available for inferring the nonconvex unknown function of interest. The framework introduces information-theoretic metrics which quantify model complexity and are used in a multi-objective formulation of the function estimation problem. The intricate relationship between information obtained through observations and model complexity is investigated. The framework is applied to the hyperparameter selection problem in Gaussian Process Regression. As a result of its generality, the framework introduced is applicable to a variety of settings and practical problems with information limitations such as channel estimation, black-box optimisation, and dual control.

[1]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[2]  Björn Wittenmark,et al.  Adaptive Dual Control Methods: An Overview , 1995 .

[3]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[4]  Rob A. Rutenbar,et al.  Simulated annealing algorithms: an overview , 1989, IEEE Circuits and Devices Magazine.

[5]  Tansu Alpcan,et al.  A framework for optimization under limited information , 2011, J. Glob. Optim..

[6]  TANSU ALPCAN,et al.  A Risk-Based Approach to Optimisation under Limited Information , 2012 .

[7]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[8]  R.N. Bracewell,et al.  Signal analysis , 1978, Proceedings of the IEEE.

[9]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Graduate Texts in Computer Science.

[10]  Iain Murray Introduction To Gaussian Processes , 2008 .

[11]  M. Peruggia Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.) , 2003 .

[12]  Kellen Petersen August Real Analysis , 2009 .

[13]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[14]  R. Tempo,et al.  Randomized Algorithms for Analysis and Control of Uncertain Systems , 2004 .

[15]  Paulo Jorge S. G. Ferreira,et al.  Nonuniform sampling of nonbandlimited signals , 1995, IEEE Signal Processing Letters.

[16]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[17]  P. L. Butzer,et al.  A Sampling Theorem for Duration-Limited Functions with Error Estimates , 1977, Inf. Control..