Learning Approximately Objective Priors

Informative Bayesian priors are often difficult to elicit, and when this is the case, modelers usually turn to noninformative or objective priors. However, objective priors such as the Jeffreys and reference priors are not tractable to derive for many models of interest. We address this issue by proposing techniques for learning reference prior approximations: we select a parametric family and optimize a black-box lower bound on the reference prior objective to find the member of the family that serves as a good approximation. We experimentally demonstrate the method's effectiveness by recovering Jeffreys priors and learning the Variational Autoencoder's reference prior.

[1]  M. Newton,et al.  Estimating the Integrated Likelihood via Posterior Simulation Using the Harmonic Mean Identity , 2006 .

[2]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[3]  Larry A. Wasserman,et al.  Iterative Markov Chain Monte Carlo Computation of Reference Priors and Minimax Risk , 2001, UAI.

[4]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[5]  James O. Berger,et al.  Overall Objective Priors , 2015, 1504.02689.

[6]  H. Jeffreys.  An invariant form for the prior probability in estimation problems , 1946, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[7]  Qiang Liu,et al.  Two Methods for Wild Variational Inference , 2016, 1612.00081.

[8]  Nozer D. Singpurwalla,et al.  Non-informative priors do not exist A dialogue with José M. Bernardo , 1997 .

[9]  Creasy Problem,et al.  Reference Posterior Distributions for Bayesian Inference , 1979 .

[10]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[11]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[12]  Christian P. Robert,et al.  The Bayesian choice , 1994 .

[13]  Dilin Wang,et al.  Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm , 2016, NIPS.

[14]  J. Ghosh,et al.  On Divergence Measures Leading to Jeffreys and Other Reference Priors , 2014 .

[15]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[16]  J. Bernardo.  Reference Analysis , 2005 .

[17]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Dustin Tran,et al.  Operator Variational Inference , 2016, NIPS.

[20]  Richard E. Turner,et al.  Variational Inference with Rényi Divergence , 2016, ArXiv.

[21]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[22]  J. Bernardo,et al.  THE FORMAL DEFINITION OF REFERENCE PRIORS , 2009, 0904.0156.