Rate-distortion optimization guided autoencoder for isometric embedding in Euclidean latent space

To analyze high-dimensional and complex data in the real world, generative model approach of machine learning aims to reduce the dimension and acquire a probabilistic model of the data. For this purpose, deep-autoencoder based generative models such as variational autoencoder (VAE) have been proposed. However, in previous works, the scale of metrics between the real and the reduced-dimensional space(latent space) is not well-controlled. Therefore, the quantitative impact of the latent variable on real data is unclear. In the end, the probability distribution function (PDF) in the real space cannot be estimated from that of the latent space accurately. To overcome this problem, we propose Rate-Distortion Optimization guided autoencoder. We show our method has the following properties theoretically and experimentally: (i) the columns of Jacobian matrix between two spaces is constantly-scaled orthonormal system and data can be embedded in a Euclidean space isometrically; (ii) the PDF of the latent space is proportional to that of the real space. Furthermore, to verify the usefulness in the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.

[1]  Jing Zhou,et al.  Multi-scale and Context-adaptive Entropy Model for Image Compression , 2019, CVPR Workshops.

[2]  Marina Meila,et al.  Nearly Isometric Embedding by Relaxation , 2016, NIPS.

[3]  P. Thomas Fletcher,et al.  The Riemannian Geometry of Deep Generative Models , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Valero Laparra,et al.  Density Modeling of Images using a Generalized Normalization Transformation , 2015, ICLR.

[8]  Dale J. Poirier,et al.  Intermediate Statistics and Econometrics: A Comparative Approach , 1995 .

[9]  Jia-Xing Hong,et al.  Isometric Embedding of Riemannian Manifolds in Euclidean Spaces , 2006 .

[10]  Georg Martius,et al.  Variational Autoencoders Pursue PCA Directions (by Accident) , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Alexander M. Bronstein,et al.  DIMAL: Deep Isometric Manifold Learning Using Sparse Geodesic Sampling , 2017, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[12]  Yifan Guo,et al.  A Unified Unsupervised Gaussian Mixture Variational Autoencoder for High Dimensional Outlier Detection , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[13]  Ryan P. Adams,et al.  Composing graphical models with neural networks for structured representations and fast inference , 2016, NIPS.

[14]  Bo Zong,et al.  Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection , 2018, ICLR.

[15]  Alexander A. Alemi,et al.  Fixing a Broken ELBO , 2017, ICML.

[16]  Chuan Sheng Foo,et al.  Adversarially Learned Anomaly Detection , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[17]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[18]  Huachun Tan,et al.  Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering , 2016, IJCAI.

[19]  Gary J. Sullivan,et al.  Rate-distortion optimization for video compression , 1998, IEEE Signal Process. Mag..

[20]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[21]  Dennis S. Bernstein,et al.  Scalar, Vector, and Matrix Mathematics: Theory, Facts, and Formulas - Revised and Expanded Edition , 2018 .

[22]  Raghavendra Chalapathy University of Sydney,et al.  Deep Learning for Anomaly Detection: A Survey , 2019, ArXiv.

[23]  I. Hassan.  Embedded , 2005, The Cyber Security Handbook.

[24]  Rob Brekelmans,et al.  Exact Rate-Distortion in Autoencoders via Echo Noise , 2019, NeurIPS.

[25]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[26]  Stefano Ermon,et al.  InfoVAE: Balancing Learning and Inference in Variational Autoencoders , 2019, AAAI.

[27]  K. R. Rao,et al.  The Transform and Data Compression Handbook , 2000 .

[28]  Xueyan Jiang,et al.  Metrics for Deep Generative Models , 2017, AISTATS.

[29]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[30]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[31]  Zhijian Ou,et al.  Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detection , 2018 .

[32]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[33]  Dinh Van Huynh,et al.  Algebra and Its Applications , 2006 .

[34]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[35]  Zhijian Ou,et al.  Learning Neural Random Fields with Inclusive Auxiliary Generators , 2018, ArXiv.

[36]  Cong Geng,et al.  Uniform Interpolation Constrained Geodesic Learning on Data Manifold , 2020, ArXiv.

[37]  Vivek K. Goyal,et al.  Theoretical foundations of transform coding , 2001, IEEE Signal Process. Mag..

[38]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[39]  David Minnen,et al.  Variational image compression with a scale hyperprior , 2018, ICLR.

[40]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[41]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Max Welling,et al.  Semi-supervised Learning with Deep Generative Models , 2014, NIPS.

[43]  Olivier Bachem,et al.  Recent Advances in Autoencoder-Based Representation Learning , 2018, ArXiv.

[44]  Oriol Vinyals,et al.  Neural Discrete Representation Learning , 2017, NIPS.