论文信息 - i-DenseNets

i-DenseNets

We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient alternative to Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce the invertibility of the network by satisfying the Lipschitz constraint. Additionally, we extend this method by proposing a learnable concatenation, which not only improves the model performance but also indicates the importance of the concatenated representation. We demonstrate the performance of i-DenseNets and Residual Flows on toy, MNIST, and CIFAR10 data. Both i-DenseNets outperform Residual Flows evaluated in negative log-likelihood, on all considered datasets under an equal parameter budget.

Sandjai Bhulai | Jakub M. Tomczak | Yura Perugachi-Diaz

[1] Yuichi Yoshida,et al. Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[2] J. Skilling. The Eigenvalues of Mega-dimensional Matrices , 1989 .

[3] Yee Whye Teh,et al. Hybrid Models with Deep and Invertible Features , 2019, ICML.

[4] Jian Sun,et al. Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5] Kilian Q. Weinberger,et al. Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6] Richard Zemel,et al. Conditional Generative Models are not Robust , 2019, ArXiv.

[7] Samy Bengio,et al. Density estimation using Real NVP , 2016, ICLR.

[8] M. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[9] Matthias Bethge,et al. Excessive Invariance Causes Adversarial Vulnerability , 2018, ICLR.

[10] Bernhard Pfahringer,et al. Regularisation of neural networks by enforcing Lipschitz continuity , 2018, Machine Learning.

[11] David Duvenaud,et al. Residual Flows for Invertible Generative Modeling , 2019, NeurIPS.