Momentum Capsule Networks

Capsule networks are a class of neural networks that achieved promising results on many computer vision tasks. However, baseline capsule networks have failed to reach state-of-the-art results on more complex datasets due to the high computation and memory requirements. We tackle this problem by proposing a new network architecture, called Momentum Capsule Network (MoCapsNet). MoCapsNets are inspired by Momentum ResNets, a type of network that applies reversible residual building blocks. Reversible networks allow for recalculating activations of the forward pass in the backpropagation algorithm, so those memory requirements can be drastically reduced. In this paper, we provide a framework on how invertible residual building blocks can be applied to capsule networks. We will show that MoCapsNet beats the accuracy of baseline capsule networks on MNIST, SVHN and CIFAR-10 while using considerably less memory. The source code is available on https://github.com/moejoe95/MoCapsNet .

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Geoffrey E. Hinton,et al.  Matrix capsules with EM routing , 2018, ICLR.

[3]  Less Wright,et al.  Ranger21: a synergistic deep learning optimizer , 2021, ArXiv.

[4]  Martin Wistuba,et al.  A Survey on Neural Architecture Search , 2019, ArXiv.

[5]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[6]  Raquel Urtasun,et al.  The Reversible Residual Network: Backpropagation Without Storing Activations , 2017, NIPS.

[7]  Haixia Xu,et al.  Dense capsule networks with fewer parameters , 2021, Soft Comput..

[8]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[9]  Eldad Haber,et al.  Reversible Architectures for Arbitrarily Deep Residual Neural Networks , 2017, AAAI.

[10]  Yang Jin,et al.  Capsule Network Performance on Complex Data , 2017, ArXiv.

[11]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[12]  Stefanos Kollias,et al.  Capsule Routing via Variational Bayes , 2019, AAAI.

[13]  Feifei Lee,et al.  RS-CapsNet: An Advanced Capsule Network , 2020, IEEE Access.

[14]  Chen Xu,et al.  MS-CapsNet: A Novel Multi-Scale Capsule Network , 2018, IEEE Signal Processing Letters.

[15]  Pin Wan,et al.  ResCaps: an improved capsule network and its application in ultrasonic image classification of thyroid papillary carcinoma , 2021, Complex & Intelligent Systems.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Mathieu Blondel,et al.  Momentum Residual Neural Networks , 2021, ICML.

[19]  Y. LeCun,et al.  Learning methods for generic object recognition with invariance to pose and lighting , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[20]  David Peer,et al.  Training Deep Capsule Networks with Residual Connections , 2021, ICANN.

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Jie Li,et al.  Group Reconstruction and Max-Pooling Residual Capsule Network , 2019, IJCAI.

[23]  Jin Liu,et al.  Multi-Lane Capsule Network for Classifying Images With Complex Background , 2020, IEEE Access.

[24]  Sebastian Stabinger,et al.  Limitation of capsule networks , 2019, Pattern Recognit. Lett..

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[27]  Sepp Hochreiter,et al.  Untersuchungen zu dynamischen neuronalen Netzen , 1991 .

[28]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.