Bayesian Model Comparison With the g-Prior

Model comparison and selection is an important problem in many model-based signal processing applications. Often, very simple information criteria such as the Akaike information criterion or the Bayesian information criterion are used despite their shortcomings. Compared to these methods, Djuric's asymptotic MAP rule was an improvement, and in this paper, we extend the work by Djuric in several ways. Specifically, we consider the elicitation of proper prior distributions, treat the case of real- and complex-valued data simultaneously in a Bayesian framework similar to that considered by Djuric, and develop new model selection rules for a regression model containing both linear and non-linear parameters. Moreover, we use this framework to give a new interpretation of the popular information criteria and relate their performance to the signal-to-noise ratio of the data. By use of simulations, we also demonstrate that our proposed model comparison and selection rules outperform the traditional information criteria both in terms of detecting the true model and in terms of predicting unobserved data. The simulation code is available online.

[1]  Petar M. Djuric,et al.  A model selection rule for sinusoids in white Gaussian noise , 1996, IEEE Trans. Signal Process..

[2]  E. T. Jaynes,et al.  Bayesian Spectrum and Chirp Analysis , 1987 .

[3]  S. Koreisha,et al.  A Comparison among Identification Procedures for Autoregressive Moving Average Models , 1991 .

[4]  Anil K. Jain,et al.  Simultaneous feature selection and clustering using mixture models , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  J. Rissanen.  Estimation of structure by minimum description length , 1982 .

[6]  A. Deltell,et al.  Objective Bayes Criteria for Variable Selection , 2011 .

[7]  Roland Badeau,et al.  A new perturbation analysis for signal enumeration in rotational invariance techniques , 2006, IEEE Transactions on Signal Processing.

[8]  Mónica F. Bugallo,et al.  Joint Model Selection and Parameter Estimation by Population Monte Carlo Simulation , 2010, IEEE Journal of Selected Topics in Signal Processing.

[9]  Andrew T. A. Wood,et al.  Laplace approximations for hypergeometric functions with matrix argument , 2002 .

[10]  L. M. Berliner,et al.  Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems , 1989 .

[11]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[12]  M. Davy,et al.  Bayesian analysis of polyphonic western tonal music. , 2006, The Journal of the Acoustical Society of America.

[13]  James O. Berger,et al.  Objective Bayesian Methods for Model Selection: Introduction and Comparison , 2001 .

[14]  A. Brix.  Bayesian Data Analysis, 2nd edn , 2005 .

[15]  B. G. Quinn,et al.  The determination of the order of an autoregression , 1979 .

[16]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[17]  R. Jaszczak,et al.  Parameter estimation of finite mixtures using the EM algorithm and information criteria with application to medical image processing , 1992 .

[18]  Dean Phillips Foster,et al.  Calibration and Empirical Bayes Variable Selection , 1997 .

[19]  Carl E. Rasmussen,et al.  The Infinite Gaussian Mixture Model , 1999, NIPS.

[20]  Mads Grcesboll Christensen,et al.  Joint direction-of-arrival and order estimation in compressed sensing using angles between subspaces , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[21]  Søren Holdt Jensen,et al.  Default Bayesian Estimation of the Fundamental Frequency , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[22]  E. Hannan,et al.  DETERMINING THE NUMBER OF TERMS IN A TRIGONOMETRIC REGRESSION , 1994 .

[23]  ByoungSeon Choi,et al.  Arma Model Identification , 1992 .

[24]  S. Chib,et al.  Marginal Likelihood From the Metropolis–Hastings Output , 2001 .

[25]  B. G. Quinn,et al.  ESTIMATING THE NUMBER OF TERMS IN A SINUSOIDAL REGRESSION , 1989 .

[26]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[27]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[28]  Bradley P. Carlin,et al.  Markov Chain Monte Carlo Methods for Computing Bayes Factors , 2001 .

[29]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[30]  Clifford M. Hurvich,et al.  A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTION , 1993 .

[31]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[32]  Thomas Kailath,et al.  Detection of signals by information theoretic criteria , 1985, IEEE Trans. Acoust. Speech Signal Process..

[33]  H. Akaike.  A new look at the statistical model identification , 1974 .

[34]  Petre Stoica,et al.  On the Proper Forms of BIC for Model Order Selection , 2012, IEEE Transactions on Signal Processing.

[35]  Petar M. Djuric,et al.  Asymptotic MAP criteria for model selection , 1998, IEEE Trans. Signal Process..

[36]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[37]  Edwin T. Jaynes.  Prior Probabilities , 2010, Encyclopedia of Machine Learning.

[38]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[39]  E. Hannan,et al.  The determination of optimum structures for the state space representation of multivariate stochastic processes , 1982 .

[40]  Andreas Jakobsson,et al.  Multi-Pitch Estimation , 2009, Multi-Pitch Estimation.

[41]  G. Kitagawa,et al.  Generalised information criteria in model selection , 1996 .

[42]  M. Tribus,et al.  Probability theory: the logic of science , 2003 .

[43]  Keith Q. T. Zhang,et al.  Information theoretic criteria for the determination of the number of signals in spatially correlated noise , 1993, IEEE Trans. Signal Process..

[44]  Kenneth P. Camilleri,et al.  Order Estimation of Multivariate ARMA Models , 2010, IEEE Journal of Selected Topics in Signal Processing.

[45]  G. Box,et al.  Empirical Model-Building and Response Surfaces. , 1990 .

[46]  G. L. Bretthorst.  The Near-Irrelevance of Sampling Frequency Distributions , 1999 .

[47]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[48]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[49]  Jian Li,et al.  Efficient mixed-spectrum estimation with applications to target feature extraction , 1995, Conference Record of The Twenty-Ninth Asilomar Conference on Signals, Systems and Computers.

[50]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[51]  S. Chib.  Marginal Likelihood from the Gibbs Output , 1995 .

[52]  Steven Kay,et al.  Inconsistency of the MDL: On the Performance of Model Order Selection Criteria With Increasing Signal-to-Noise Ratio , 2011, IEEE Transactions on Signal Processing.

[53]  D. Mitchell Wilkes,et al.  ARMA model order estimation based on the eigenvalues of the covariance matrix , 1993, IEEE Trans. Signal Process..

[54]  Jorma Rissanen,et al.  A Predictive Least-Squares Principle , 1986 .

[55]  Petar M. Djuric,et al.  Model selection based on Bayesian predictive densities and multiple data records , 1994, IEEE Trans. Signal Process..

[56]  M. Steel,et al.  Benchmark Priors for Bayesian Model Averaging , 2001 .

[57]  Rodney W. Strachan,et al.  Improper priors with well defined Bayes Factors , 2004 .

[58]  Christophe Andrieu,et al.  Joint Bayesian model selection and estimation of noisy sinusoids via reversible jump MCMC , 1999, IEEE Trans. Signal Process..

[59]  A. Doucet,et al.  Computational Advances for and from Bayesian Analysis , 2004 .

[60]  Lynne Seymour,et al.  Institute of Mathematical Statistics LECTURE NOTES ? MONOGRAPH SERIES , 2016 .

[61]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[62]  Edward I. George,et al.  Empirical Bayes vs. Fully Bayes Variable Selection , 2008 .

[63]  Leonhard Held,et al.  Hyper-$g$ priors for generalized linear models , 2010, 1008.1550.

[64]  Søren Holdt Jensen,et al.  Bayesian model comparison and the BIC for regression models , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[65]  M. Clyde,et al.  Model Uncertainty , 2003 .

[66]  Yuhong Yang,et al.  Information Theory, Inference, and Learning Algorithms , 2005 .

[67]  Petre Stoica,et al.  Spectral Analysis of Signals , 2009 .

[68]  L. Shields.  The logic of science. , 2007, Paediatric nursing.

[69]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[70]  Jian Li,et al.  Multi-model approach to model selection , 2004, Digit. Signal Process..

[71]  Y. Selen,et al.  Model-order selection: a review of information criterion rules , 2004, IEEE Signal Processing Magazine.

[72]  I. S. Gradshteyn,et al.  Table of Integrals, Series, and Products , 1976 .

[73]  A. Doucet,et al.  Reversible Jump Markov Chain Monte Carlo Strategies for Bayesian Model Selection in Autoregressive Processes , 2004, Journal of Time Series Analysis.

[74]  Andreas Jakobsson,et al.  Sinusoidal Order Estimation Using Angles between Subspaces , 2009, EURASIP J. Adv. Signal Process..

[75]  Kerstin Vogler,et al.  Table Of Integrals Series And Products , 2016 .

[76]  A. Zellner,et al.  Posterior odds ratios for selected regression hypotheses , 1980 .

[77]  Sabine Van Huffel,et al.  A Shift Invariance-Based Order-Selection Technique for Exponential Data Modelling , 2007, IEEE Signal Processing Letters.

[78]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[79]  Christian P. Robert,et al.  The Bayesian choice : from decision-theoretic foundations to computational implementation , 2007 .

[80]  George E. P. Box,et al.  Empirical Model‐Building and Response Surfaces , 1988 .