IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression

In this paper we analyse and improve integer discrete flows for lossless compression. Integer discrete flows are a recently proposed class of models that learn invertible transformations for integer-valued random variables. Due to its discrete nature, they can be combined in a straightforward manner with entropy coding schemes for lossless compression without the need for bits-back coding. We discuss the potential difference in flexibility between invertible flows for discrete random variables and flows for continuous random variables and show that (integer) discrete flows are more flexible than previously claimed. We furthermore investigate the influence of quantization operators on optimization and gradient bias in integer discrete flows. Finally, we introduce modifications to the architecture to improve the performance of this model class for lossless compression.

[1]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[2]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[3]  Iain Murray,et al.  Neural Spline Flows , 2019, NeurIPS.

[4]  Luc Van Gool,et al.  Learning Better Lossless Compression Using Lossy Compression , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Will Grathwohl.  Scalable Reversible Generative Models with Free-form Continuous Dynamics , 2018 .

[6]  Michael W. Marcellin,et al.  JPEG2000 - image compression fundamentals, standards and practice , 2013, The Kluwer international series in engineering and computer science.

[7]  Ole Winther,et al.  Ladder Variational Autoencoders , 2016, NIPS.

[8]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[9]  Pieter Abbeel,et al.  Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables , 2019, ICML.

[10]  Kumar Krishna Agrawal,et al.  Discrete Flows: Invertible Generative Models of Discrete Data , 2019, DGS@ICLR.

[11]  Peter Deutsch,et al.  DEFLATE Compressed Data Format Specification version 1.3 , 1996, RFC.

[12]  Garrison W. Cottrell,et al.  ReZero is All You Need: Fast Convergence at Large Depth , 2020, UAI.

[13]  Hugo Larochelle,et al.  RNADE: The real-valued neural autoregressive density-estimator , 2013, NIPS.

[14]  James Townsend.  A tutorial on the range variant of asymmetric numeral systems , 2020, ArXiv.

[15]  Hao Li,et al.  Visualizing the Loss Landscape of Neural Nets , 2017, NeurIPS.

[16]  Matthias Bethge,et al.  A note on the evaluation of generative models , 2015, ICLR.

[17]  Sungwon Kim,et al.  FloWaveNet : A Generative Flow for Raw Audio , 2018, ICML.

[18]  Ole Winther,et al.  BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling , 2019, NeurIPS.

[19]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[20]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[21]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[22]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Quoc V. Le,et al.  Searching for Activation Functions , 2018, arXiv.

[24]  Philipp Krähenbühl,et al.  Lossless Image Compression through Super-Resolution , 2020, ArXiv.

[25]  Jarek Duda,et al.  Asymmetric numeral systems , 2009, ArXiv.

[26]  E. Tabak,et al.  DENSITY ESTIMATION BY DUAL ASCENT OF THE LOG-LIKELIHOOD ∗ , 2010 .

[27]  Pieter Abbeel,et al.  Compression with Flows via Local Bits-Back Coding , 2019, NeurIPS.

[28]  Max Welling,et al.  Emerging Convolutions for Generative Normalizing Flows , 2019, ICML.

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Xi Chen,et al.  PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications , 2017, ICLR.

[31]  Jarek Duda,et al.  Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding , 2013, 1311.2540.

[32]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[33]  Sergey Levine,et al.  VideoFlow: A Flow-Based Generative Model for Video , 2019, ArXiv.

[34]  David Barber,et al.  HiLLoC: Lossless Image Compression with Hierarchical Latent Variable Models , 2019, ICLR.

[35]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[36]  Yoshua Bengio,et al.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , 2013, ArXiv.

[37]  Emiel Hoogeboom,et al.  Integer Discrete Flows and Lossless Compression , 2019, NeurIPS.

[38]  Paul Vicol,et al.  Understanding and mitigating exploding inverses in invertible neural networks , 2020, AISTATS.

[39]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[40]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[41]  David Barber,et al.  Practical Lossless Compression with Latent Variables using Bits Back Coding , 2019, ICLR.

[42]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[43]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[44]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[45]  Pieter Abbeel,et al.  Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design , 2019, ICML.

[46]  E. Tabak,et al.  A Family of Nonparametric Density Estimation Algorithms , 2013 .

[47]  Thomas Boutell,et al.  PNG (Portable Network Graphics) Specification Version 1.0 , 1997, RFC.

[48]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..

[49]  Luc Van Gool,et al.  Practical Full Resolution Learned Lossless Image Compression , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Jon Sneyers,et al.  FLIF: Free lossless image format based on MANIAC compression , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[51]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .