Implicit Maximum Likelihood Estimation

Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any derived quantities, but can be shown to be equivalent to maximizing likelihood under some conditions. Our result holds in the non-asymptotic parametric setting, where both the capacity of the model and the number of data examples are finite. We also demonstrate encouraging experimental results.

[1]  F. Y. Edgeworth.  On the Probable Errors of Frequency-Constants , 1908 .

[2]  R. Fisher.  001: On an Absolute Criterion for Fitting Frequency Curves. , 1912 .

[3]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[4]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[5]  J. Besag.  Statistical Analysis of Non-Lattice Data , 1975 .

[6]  P. Diggle,et al.  Monte Carlo Methods of Inference for Implicit Statistical Models , 1984 .

[7]  Eitan Zemel,et al.  Nash and correlated equilibria: Some complexity considerations , 1989 .

[8]  J. Urgen Schmidhuber,et al.  Learning Factorial Codes by Predictability Minimization , 1992 .

[9]  Radford M. Neal.  Connectionist Learning of Belief Networks , 1992, Artif. Intell..

[10]  A. Müller.  Integral Probability Metrics and Their Generating Classes of Functions , 1997, Advances in Applied Probability.

[11]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[12]  Samy Bengio,et al.  Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks , 1999, NIPS.

[13]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[14]  Geoffrey E. Hinton,et al.  A New Learning Algorithm for Mean Field Boltzmann Machines , 2002, ICANN.

[15]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[16]  Aapo Hyvärinen,et al.  Estimation of Non-Normalized Statistical Models by Score Matching , 2005, J. Mach. Learn. Res..

[17]  Bernhard Schölkopf,et al.  A Kernel Method for the Two-Sample-Problem , 2006, NIPS.

[18]  Zhuowen Tu,et al.  Learning Generative Models via Discriminative Approaches , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Aapo Hyvärinen,et al.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models , 2010, AISTATS.

[20]  Hugo Larochelle,et al.  The Neural Autoregressive Distribution Estimator , 2011, AISTATS.

[21]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[22]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[23]  Yoshua Bengio,et al.  Deep Generative Stochastic Networks Trainable by Backprop , 2013, ICML.

[24]  K. Perez.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment , 2014 .

[25]  Daan Wierstra,et al.  Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models , 2014, ArXiv.

[26]  Ferenc Huszar,et al.  How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? , 2015, ArXiv.

[27]  Zoubin Ghahramani,et al.  Training generative neural networks via Maximum Mean Discrepancy optimization , 2015, UAI.

[28]  Richard S. Zemel,et al.  Generative Moment Matching Networks , 2015, ICML.

[29]  Jitendra Malik,et al.  Fast k-Nearest Neighbour Search via Dynamic Continuous Indexing , 2015, ICML.

[30]  Shakir Mohamed,et al.  Learning in Implicit Generative Models , 2016, ArXiv.

[31]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[32]  Matthias Bethge,et al.  A note on the evaluation of generative models , 2015, ICLR.

[33]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[34]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[35]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[36]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[37]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[38]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[39]  Yoshua Bengio,et al.  Boundary-Seeking Generative Adversarial Networks , 2017, ICLR 2017.

[40]  Stefano Ermon,et al.  Flow-GAN: Bridging implicit and prescribed learning in generative models , 2017, ArXiv.

[41]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[42]  Jitendra Malik,et al.  Fast k-Nearest Neighbour Search via Prioritized DCI , 2017, ICML.

[43]  Sebastian Nowozin,et al.  The Numerics of GANs , 2017, NIPS.

[44]  Yingyu Liang,et al.  Generalization and Equilibrium in Generative Adversarial Nets (GANs) , 2017, ICML.

[45]  Ruslan Salakhutdinov,et al.  On the Quantitative Analysis of Decoder-Based Generative Models , 2016, ICLR.

[46]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[47]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[48]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[49]  Richard S. Zemel,et al.  Dualing GANs , 2017, NIPS.

[50]  Yi Zhang,et al.  Do GANs actually learn the distribution? An empirical study , 2017, ArXiv.

[51]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.

[52]  Mathieu Sinn,et al.  Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training , 2017, AISTATS.

[53]  Ritabrata Dutta,et al.  Likelihood-free inference via classification , 2014, Stat. Comput..

[54]  Stefano Ermon,et al.  Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models , 2017, AAAI.