Neural Approximate Sufficient Statistics for Implicit Models

We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of likelihood function is intractable but sampling / simulating data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representation of the data. This representation is computed by a deep neural network trained by a joint statistic-posterior learning strategy. We apply our approach to both traditional approximate Bayesian computation (ABC) and recent neural likelihood approaches, boosting their performance on a range of tasks.

[1]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[2]  P. Diggle,et al.  Monte Carlo Methods of Inference for Implicit Statistical Models , 1984 .

[3]  Joshua B. Tenenbaum,et al.  Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs , 2013, NIPS.

[4]  Jessica T Davis,et al.  The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak , 2020, Science.

[5]  Iain Murray,et al.  Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation , 2016, 1605.06376.

[6]  Sergey Levine,et al.  Wasserstein Dependency Measure for Representation Learning , 2019, NeurIPS.

[7]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[8]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[9]  Iain Murray,et al.  Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation , 2016 .

[10]  David S. Greenberg,et al.  Automatic Posterior Transformation for Likelihood-Free Inference , 2019, ICML.

[11]  M. Gutmann,et al.  Fundamentals and Recent Developments in Approximate Bayesian Computation , 2016, Systematic biology.

[12]  Michael U. Gutmann,et al.  Dynamic Likelihood-free Inference via Ratio Estimation (DIRE) , 2018, ArXiv.

[13]  Alberto D. Pascual-Montano,et al.  A survey of dimensionality reduction techniques , 2014, ArXiv.

[14]  Maria L. Rizzo,et al.  Partial Distance Correlation with Methods for Dissimilarities , 2013, 1310.2926.

[15]  Ohad Shamir,et al.  Learning and generalization with the information bottleneck , 2008, Theor. Comput. Sci..

[16]  Benjamin Dan Wandelt,et al.  Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology , 2018, 1801.01497.

[17]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[18]  Gilles Louppe,et al.  Likelihood-free MCMC with Amortized Approximate Ratio Estimators , 2019, ICML.

[19]  Peter Skands,et al.  A brief introduction to PYTHIA 8.1 , 2007, Comput. Phys. Commun..

[20]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[21]  Jukka Corander,et al.  Likelihood-Free Inference by Ratio Estimation , 2016, Bayesian Analysis.

[22]  Michael U. Gutmann,et al.  Adaptive Gaussian Copula ABC , 2019, AISTATS.

[23]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[24]  Yun S. Song,et al.  A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks , 2018, bioRxiv.

[25]  Iain Murray,et al.  On Contrastive Learning for Likelihood-free Inference , 2020, ICML.

[26]  Gilles Louppe,et al.  Approximating Likelihood Ratios with Calibrated Discriminative Classifiers , 2015, 1506.02169.

[27]  Ravi Bansal,et al.  Risks for the Long Run: A Potential Resolution of Asset Pricing Puzzles , 2000 .

[28]  Robert Leenders,et al.  Hamiltonian ABC , 2015, UAI.

[29]  Ryan P. Adams,et al.  High-Dimensional Probability Estimation with Deep Density Models , 2013, ArXiv.

[30]  S. Wood.  Statistical inference for noisy nonlinear ecological dynamic systems , 2010, Nature.

[31]  Jie Li,et al.  A survey of dimensionality reduction techniques based on random projection , 2017, ArXiv.

[32]  Paul Marjoram,et al.  Markov chain Monte Carlo without likelihoods , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[33]  Martin J. Wainwright,et al.  Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization , 2008, IEEE Transactions on Information Theory.

[34]  Jes Frellsen,et al.  Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate Bayesian Computation , 2019, ICML.

[35]  Iain Murray,et al.  Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows , 2018, AISTATS.

[36]  Jakob H. Macke,et al.  Likelihood-free inference with emulator networks , 2018, AABI.

[37]  Jakob H. Macke,et al.  Flexible statistical inference for mechanistic models of neural dynamics , 2017, NIPS.

[38]  M. Feldman,et al.  Population growth of human Y chromosomes: a study of Y chromosome microsatellites. , 1999, Molecular biology and evolution.

[39]  Gilles Louppe,et al.  Mining gold from implicit models to improve likelihood-free inference , 2018, Proceedings of the National Academy of Sciences.

[40]  Zenglin Xu,et al.  Mutual Information Gradient Estimation for Representation Learning , 2020, ICLR.

[41]  Paul Fearnhead,et al.  Constructing summary statistics for approximate Bayesian computation: semi‐automatic approximate Bayesian computation , 2012 .

[42]  W. M. Wood-Vasey,et al.  LIKELIHOOD-FREE COSMOLOGICAL INFERENCE WITH TYPE Ia SUPERNOVAE: APPROXIMATE BAYESIAN COMPUTATION FOR A COMPLETE TREATMENT OF UNCERTAINTY , 2012, 1206.2563.

[43]  Aaron C. Courville,et al.  MINE: Mutual Information Neural Estimation , 2018, ArXiv.

[44]  Heiga Zen,et al.  Parallel WaveNet: Fast High-Fidelity Speech Synthesis , 2017, ICML.

[45]  A. Pettitt,et al.  Approximate Bayesian computation using indirect inference , 2011 .

[46]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[47]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[48]  Bai Jiang,et al.  Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network , 2015, 1510.02175.

[49]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[50]  Aki Vehtari,et al.  Gaussian process modelling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria , 2016, The Annals of Applied Statistics.

[51]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.