Likelihood-free inference via classification

Increasingly complex generative models are being used across disciplines as they allow for realistic characterization of data, but a common difficulty with them is the prohibitively large computational cost to evaluate the likelihood function and thus to perform likelihood-based statistical inference. A likelihood-free inference framework has emerged where the parameters are identified by finding values that yield simulated data resembling the observed data. While widely applicable, a major difficulty in this framework is how to measure the discrepancy between the simulated and observed data. Transforming the original problem into a problem of classifying the data into simulated versus observed, we find that classification accuracy can be used to assess the discrepancy. The complete arsenal of classification methods becomes thereby available for inference of intractable generative models. We validate our approach using theory and simulations for both point estimation and Bayesian inference, and demonstrate its use on real data by inferring an individual-based epidemiological model for bacterial infections in child care centers.

[1]  Soumendu Sundar Mukherjee Weak convergence and empirical processes , 2019 .

[2]  David Pollard,et al.  A User's Guide to Measure Theoretic Probability by David Pollard , 2001 .

[3]  P. Donnelly,et al.  Inferring coalescence times from DNA sequence data. , 1997, Genetics.

[4]  Larry Wasserman,et al.  All of Statistics , 2004 .

[5]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[6]  M. Feldman,et al.  Population growth of human Y chromosomes: a study of Y chromosome microsatellites. , 1999, Molecular biology and evolution.

[7]  A. V. D. Vaart,et al.  Asymptotic Statistics: Frontmatter , 1998 .

[8]  Aapo Hyvärinen,et al.  Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics , 2012, J. Mach. Learn. Res..

[9]  Aapo Hyvärinen,et al.  A Family of Computationally E cient and Simple Estimators for Unnormalized Statistical Models , 2010, UAI.

[10]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  R. Plevin,et al.  Approximate Bayesian Computation in Evolution and Ecology , 2011 .

[13]  A. N. Pettitt,et al.  Approximate Bayesian Computation for astronomical model analysis: a case study in galaxy demographics and morphological transformation at high redshift , 2012, 1202.1426.

[14]  Paul Marjoram,et al.  Markov chain Monte Carlo without likelihoods , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[15]  P. Diggle,et al.  Monte Carlo Methods of Inference for Implicit Statistical Models , 1984 .

[16]  Andreas Huth,et al.  Statistical inference for stochastic simulation models--theory and application. , 2011, Ecology letters.

[17]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[18]  Jean-Marie Cornuet,et al.  ABC model choice via random forests , 2014, 1406.6288.

[19]  Nicolas Chopin,et al.  The Poisson transform for unnormalised statistical models , 2014, Stat. Comput..

[20]  Mats Gyllenberg,et al.  Estimating the Transmission Dynamics of Streptococcus pneumoniae from Strain Prevalence Data , 2013, Biometrics.

[21]  A. Futschik,et al.  A Novel Approach for Choosing Summary Statistics in Approximate Bayesian Computation , 2012, Genetics.

[22]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[23]  Junichiro Hirayama,et al.  Bregman divergence as general framework to estimate unnormalized statistical models , 2011, UAI.

[24]  Michael U. Gutmann,et al.  Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models , 2015, J. Mach. Learn. Res..

[25]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  D. McFadden A Method of Simulated Moments for Estimation of Discrete Response Models Without Numerical Integration , 1989 .

[28]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[29]  D. Balding,et al.  Approximate Bayesian computation in population genetics. , 2002, Genetics.

[30]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[31]  Joshua B. Tenenbaum,et al.  Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs , 2013, NIPS.

[32]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[33]  D. Pollard A User's Guide to Measure Theoretic Probability by David Pollard , 2001 .

[34]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[35]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[36]  D. Caugant,et al.  Phenotypic and Genotypic Characterization of Streptococcus pneumoniae Strains Colonizing Children Attending Day-Care Centers in Norway , 2008, Journal of Clinical Microbiology.

[37]  Benjamin T. Vincent,et al.  A tutorial on Bayesian models of perception , 2015 .

[38]  Aapo Hyvärinen,et al.  Estimation of unnormalized statistical models without numerical integration , 2013 .

[39]  Paul Fearnhead,et al.  Constructing summary statistics for approximate Bayesian computation: semi‐automatic approximate Bayesian computation , 2012 .

[40]  Tong Zhang Statistical behavior and consistency of classification methods based on convex risk minimization , 2003 .

[41]  J. Møller Discussion on the paper by Feranhead and Prangle , 2012 .

[42]  David J. Nott,et al.  A note on approximating ABC‐MCMC using flexible classifiers , 2014 .

[43]  M. Gutmann,et al.  Fundamentals and Recent Developments in Approximate Bayesian Computation , 2016, Systematic biology.

[44]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[45]  Long Zhu,et al.  Unsupervised Learning of Probabilistic Grammar-Markov Models for Object Categories , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  David Welch,et al.  Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems , 2009, Journal of The Royal Society Interface.

[47]  Jean-Michel Marin,et al.  Approximate Bayesian computational methods , 2011, Statistics and Computing.

[48]  L. Blume,et al.  The New Palgrave Dictionary of Economics, 2nd edition , 2008 .

[49]  A. Pettitt,et al.  Approximate Bayesian computation using indirect inference , 2011 .

[50]  D. Pollard,et al.  Simulation and the Asymptotics of Optimization Estimators , 1989 .

[51]  L. Excoffier,et al.  Efficient Approximate Bayesian Computation Coupled With Markov Chain Monte Carlo Without Likelihood , 2009, Genetics.

[52]  M. Gutmann,et al.  Likelihood-free inference by penalised logistic regression , 2016 .

[53]  Aapo Hyvärinen,et al.  A three-layer model of natural image statistics , 2013, Journal of Physiology-Paris.

[54]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.

[55]  Jukka Corander,et al.  Likelihood-free inference via classification , 2014, Statistics and Computing.