Density Deconvolution with Normalizing Flows

Density deconvolution is the task of estimating a probability density function given only noise-corrupted samples. We can fit a Gaussian mixture model to the underlying density by maximum likelihood if the noise is normally distributed, but would like to exploit the superior density estimation performance of normalizing flows and allow for arbitrary noise distributions. Since both adjustments lead to an intractable likelihood, we resort to amortized variational inference. We demonstrate some problems involved in this approach, however, experiments on real data demonstrate that flows can already out-perform Gaussian mixtures for density deconvolution.

[1]  Sam T. Roweis,et al.  Extreme deconvolution: Inferring complete distribution functions from noisy, heterogeneous and incomplete observations , 2009, 0905.2979.

[2]  Xiao Wang,et al.  Unbiased Contrastive Divergence Algorithm for Training Energy-Based Latent Variable Models , 2020, ICLR.

[3]  Peter W. Glynn,et al.  Exact estimation for Markov chain equilibrium expectations , 2014, Journal of Applied Probability.

[4]  Jianqing Fan.  On the Optimal Rates of Convergence for Nonparametric Deconvolution Problems , 1991 .

[5]  Boris Leistedt,et al.  Improving Gaia Parallax Precision with a Data-driven Model of Stars , 2017, The Astronomical Journal.

[6]  Oriol Vinyals,et al.  Neural Discrete Representation Learning , 2017, NIPS.

[7]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[8]  L. Devroye.  Consistent deconvolution in density estimation , 1989 .

[9]  E. Tabak,et al.  DENSITY ESTIMATION BY DUAL ASCENT OF THE LOG-LIKELIHOOD ∗ , 2010 .

[10]  David N. Spergel,et al.  Modeling the Gaia Color-Magnitude Diagram with Bayesian Neural Flows to Constrain Distance Estimates , 2019, ArXiv.

[11]  B. Vidakovic,et al.  Adaptive wavelet estimator for nonparametric density deconvolution , 1999 .

[12]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[13]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[14]  D. Rubin.  Using the SIR algorithm to simulate posterior distributions , 1988 .

[15]  Yaoliang Yu,et al.  Sum-of-Squares Polynomial Flow , 2019, ICML.

[16]  Iain Murray,et al.  Scalable Extreme Deconvolution , 2019, ArXiv.

[17]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[18]  Hugo Larochelle,et al.  RNADE: The real-valued neural autoregressive density-estimator , 2013, NIPS.

[19]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Charlie Nash,et al.  Autoregressive Energy Machines , 2019, ICML.

[22]  David Duvenaud,et al.  Inference Suboptimality in Variational Autoencoders , 2018, ICML.

[23]  Wes McKinney,et al.  Data Structures for Statistical Computing in Python , 2010, SciPy.

[24]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[25]  P. Hall,et al.  Optimal Rates of Convergence for Deconvolving a Density , 1988 .

[26]  Lars Hertel,et al.  Approximate Inference for Deep Latent Gaussian Mixtures , 2016 .

[27]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[28]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[29]  Murray Shanahan,et al.  Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders , 2016, ArXiv.

[30]  David Duvenaud,et al.  Reinterpreting Importance-Weighted Autoencoders , 2017, ICLR.

[31]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[32]  Pieter Abbeel,et al.  Variational Lossy Autoencoder , 2016, ICLR.

[33]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[34]  Robert L. Taylor,et al.  A consistent nonparametric density estimator for the deconvolution problem , 1989 .

[35]  Ryan P. Adams,et al.  SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models , 2020, ICLR.

[36]  Iain Murray,et al.  Neural Spline Flows , 2019, NeurIPS.

[37]  Raymond J. Carroll,et al.  Measurement error in nonlinear models: a modern perspective , 2006 .

[38]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[39]  Daniel Foreman-Mackey,et al.  corner.py: Scatterplot matrices in Python , 2016, J. Open Source Softw..

[40]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[41]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[42]  E. Tabak,et al.  A Family of Nonparametric Density Estimation Algorithms , 2013 .

[43]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..