Neural Approximate Sufficient Statistics for Implicit Models

We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of likelihood function is intractable but sampling / simulating data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representation of the data. This representation is computed by a deep neural network trained by a joint statistic-posterior learning strategy. We apply our approach to both traditional approximate Bayesian computation (ABC) and recent neural likelihood approaches, boosting their performance on a range of tasks.

[1]  S. Wood.  Statistical inference for noisy nonlinear ecological dynamic systems , 2010, Nature.

[2]  Jukka Corander,et al.  Likelihood-Free Inference by Ratio Estimation , 2016, Bayesian Analysis.

[3]  Jakob H. Macke,et al.  Flexible statistical inference for mechanistic models of neural dynamics , 2017, NIPS.

[4]  Michael U. Gutmann,et al.  Adaptive Gaussian Copula ABC , 2019, AISTATS.

[5]  Aaron C. Courville,et al.  MINE: Mutual Information Neural Estimation , 2018, ArXiv.

[6]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[7]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[8]  M. Feldman,et al.  Population growth of human Y chromosomes: a study of Y chromosome microsatellites. , 1999, Molecular biology and evolution.

[9]  Ohad Shamir,et al.  Learning and generalization with the information bottleneck , 2008, Theor. Comput. Sci..

[10]  Bai Jiang,et al.  Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network , 2015, 1510.02175.

[11]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[12]  Ryan P. Adams,et al.  High-Dimensional Probability Estimation with Deep Density Models , 2013, ArXiv.

[13]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[14]  Sergey Levine,et al.  Wasserstein Dependency Measure for Representation Learning , 2019, NeurIPS.

[15]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[16]  Alberto D. Pascual-Montano,et al.  A survey of dimensionality reduction techniques , 2014, ArXiv.

[17]  Gilles Louppe,et al.  Approximating Likelihood Ratios with Calibrated Discriminative Classifiers , 2015, 1506.02169.

[18]  Robert Leenders,et al.  Hamiltonian ABC , 2015, UAI.

[19]  Iain Murray,et al.  Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation , 2016, 1605.06376.

[20]  Heiga Zen,et al.  Parallel WaveNet: Fast High-Fidelity Speech Synthesis , 2017, ICML.

[21]  Gilles Louppe,et al.  Mining gold from implicit models to improve likelihood-free inference , 2018, Proceedings of the National Academy of Sciences.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Jessica T Davis,et al.  The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak , 2020, Science.

[24]  Paul Marjoram,et al.  Markov chain Monte Carlo without likelihoods , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Ravi Bansal,et al.  Risks for the Long Run: A Potential Resolution of Asset Pricing Puzzles , 2000 .

[26]  Peter Skands,et al.  A brief introduction to PYTHIA 8.1 , 2007, Comput. Phys. Commun..

[27]  Benjamin Dan Wandelt,et al.  Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology , 2018, 1801.01497.

[28]  Jes Frellsen,et al.  Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate Bayesian Computation , 2019, ICML.

[29]  Aki Vehtari,et al.  Gaussian process modelling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria , 2016, The Annals of Applied Statistics.

[30]  Jie Li,et al.  A survey of dimensionality reduction techniques based on random projection , 2017, ArXiv.

[31]  Yun S. Song,et al.  A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks , 2018, bioRxiv.

[32]  David S. Greenberg,et al.  Automatic Posterior Transformation for Likelihood-Free Inference , 2019, ICML.

[33]  Zenglin Xu,et al.  Mutual Information Gradient Estimation for Representation Learning , 2020, ICLR.

[34]  Iain Murray,et al.  Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows , 2018, AISTATS.

[35]  Joshua B. Tenenbaum,et al.  Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs , 2013, NIPS.

[36]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[37]  A. Pettitt,et al.  Approximate Bayesian computation using indirect inference , 2011 .

[38]  P. Diggle,et al.  Monte Carlo Methods of Inference for Implicit Statistical Models , 1984 .

[39]  Martin J. Wainwright,et al.  Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization , 2008, IEEE Transactions on Information Theory.

[40]  M. Gutmann,et al.  Fundamentals and Recent Developments in Approximate Bayesian Computation , 2016, Systematic biology.

[41]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[42]  Iain Murray,et al.  On Contrastive Learning for Likelihood-free Inference , 2020, ICML.

[43]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[44]  Gilles Louppe,et al.  Likelihood-free MCMC with Amortized Approximate Ratio Estimators , 2019, ICML.

[45]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[46]  W. M. Wood-Vasey,et al.  LIKELIHOOD-FREE COSMOLOGICAL INFERENCE WITH TYPE Ia SUPERNOVAE: APPROXIMATE BAYESIAN COMPUTATION FOR A COMPLETE TREATMENT OF UNCERTAINTY , 2012, 1206.2563.

[47]  Michael U. Gutmann,et al.  Dynamic Likelihood-free Inference via Ratio Estimation (DIRE) , 2018, ArXiv.

[48]  Paul Fearnhead,et al.  Constructing summary statistics for approximate Bayesian computation: semi‐automatic approximate Bayesian computation , 2012 .

[49]  Jakob H. Macke,et al.  Likelihood-free inference with emulator networks , 2018, AABI.