Capsule Routing via Variational Bayes

Capsule networks are a recently proposed type of neural network shown to outperform alternatives in challenging shape recognition tasks. In capsule networks, scalar neurons are replaced with capsule vectors or matrices, whose entries represent different properties of objects. The relationships between objects and their parts are learned via trainable viewpoint-invariant transformation matrices, and the presence of a given object is decided by the level of agreement among votes from its parts. This interaction occurs between capsule layers and is a process called routing-by-agreement. In this paper, we propose a new capsule routing algorithm derived from Variational Bayes for fitting a mixture of transforming gaussians, and show it is possible transform our capsule network into a Capsule-VAE. Our Bayesian approach addresses some of the inherent weaknesses of MLE based models such as the variance-collapse by modelling uncertainty over capsule pose parameters. We outperform the state-of-the-art on smallNORB using 50% fewer capsules than previously reported, achieve competitive performances on CIFAR-10, Fashion-MNIST, SVHN, and demonstrate significant improvement in MNIST to affNIST generalisation over previous works.

[1]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[2]  T. Villmann,et al.  Learning Vector Quantization Capsules , 2018 .

[3]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[4]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[5]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[6]  Y. LeCun,et al.  Learning methods for generic object recognition with invariance to pose and lighting , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[7]  Yee Whye Teh,et al.  Stacked Capsule Autoencoders , 2019, NeurIPS.

[8]  Abhinav Dhall,et al.  Dense and Diverse Capsule Networks: Making the Capsules Learn Better , 2018, ArXiv.

[9]  Geoffrey E. Hinton,et al.  Matrix capsules with EM routing , 2018, ICLR.

[10]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[11]  Sung-Hyun Son,et al.  Kernelized Capsule Networks , 2019, ArXiv.

[12]  Gideon Kowadlo,et al.  Sparse Unsupervised Capsules Generalize Better , 2018, ArXiv.

[13]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[14]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[15]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[16]  Marc Van Droogenbroeck,et al.  An Effective Hit-or-Miss Layer Favoring Feature Interpretation as Learned Prototypes Deformations , 2019, ArXiv.

[17]  Wei Zhao,et al.  Fast dynamic routing based on weighted kernel density estimation , 2018, Cognitive Internet of Things.

[18]  Edson Borin,et al.  The Multi-Lane Capsule Network (MLCN) , 2019, ArXiv.

[19]  Guo-Jun Qi,et al.  CapProNet: Deep Feature Learning via Orthogonal Projections onto Capsule Subspaces , 2018, NeurIPS.

[20]  Mubarak Shah,et al.  VideoCapsuleNet: A Simplified Network for Action Detection , 2018, NeurIPS.

[21]  Rohan Doshi,et al.  Pushing the Limits of Capsule Networks , 2018 .

[22]  Hyun Seo,et al.  Attention Routing Between Capsules , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Zhen Zhao,et al.  Capsule Networks with Max-Min Normalization , 2019, ArXiv.

[24]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Pascal Libuschewski,et al.  Group Equivariant Capsule Networks , 2018, NeurIPS.

[26]  Chen Xu,et al.  MS-CapsNet: A Novel Multi-Scale Capsule Network , 2018, IEEE Signal Processing Letters.

[27]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[28]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Taewon Jeong,et al.  Ladder Capsule Network , 2019, ICML.