A Triangular Network For Density Estimation

We report a triangular neural network implementation of neural autoregressive flow (NAF). Unlike many universal autoregressive density models, our design is highly modular, parameter economy, computationally efficient, and applicable to density estimation of data with high dimensions. It achieves state-of-the-art bits-per-dimension indices on MNIST and CIFAR-10 (about 1.1 and 3.7, respectively) in the category of general-purpose density estimators.

[1]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[2]  A. Izenman.  Recent Developments in Nonparametric Density Estimation , 1991 .

[3]  John E. Kolassa,et al.  Series Approximation Methods in Statistics , 1994 .

[4]  Amir F. Atiya,et al.  Neural Networks for Density Estimation , 1998, NIPS.

[5]  Ramesh A. Gopinath,et al.  Gaussianization , 2000, NIPS.

[6]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[7]  P. Deb.  Finite Mixture Models , 2008 .

[8]  Marina Velikova,et al.  Monotone and Partially Monotone Neural Networks , 2010, IEEE Transactions on Neural Networks.

[9]  Hugo Larochelle,et al.  The Neural Autoregressive Distribution Estimator , 2011, AISTATS.

[10]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[11]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[12]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[13]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[14]  Barnabás Póczos,et al.  Transformation Autoregressive Networks , 2018, ICML.

[15]  E Weinan,et al.  Monge-Ampère Flow for Generative Modeling , 2018, ArXiv.

[16]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[17]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[18]  Nicola De Cao,et al.  Block Neural Autoregressive Flow , 2019, UAI.

[19]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[20]  Antoine Wehenkel,et al.  Unconstrained Monotonic Neural Networks , 2019, BNAIC/BENELEARN.

[21]  Ivan Kobyzev,et al.  Normalizing Flows: An Introduction and Review of Current Methods , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.