Limitation of capsule networks

A recent development in deep learning groups multiple neurons to capsules such that each capsule represent an object or part of an object. Routing algorithms route the output of capsules from lower level layers to upper level layers. As we show in this paper, these routing procedures are unfit to learn some concrete but simple problems. Specifically, we provide a formal proof that the EM-routing and the routing-by-agreement algorithms cannot distinguish vectors from their negative counterpart. Therefore, a capsule network can only learn problems where the negative input represents the same class as the input itself. We support this theoretical work experimentally and show that a capsule network cannot classify the sign of a scalar with accuracy higher than chance. Methods that prevent this issue from happening are also presented in this paper. As we will show and reason, avoiding this drawback has a positive effect for the training of capsule networks.

[1]  Ranga Rodrigo,et al.  DeepCaps: Going Deeper With Capsule Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Mubarak Shah,et al.  VideoCapsuleNet: A Simplified Network for Action Detection , 2018, NeurIPS.

[3]  Geoffrey E. Hinton,et al.  Matrix capsules with EM routing , 2018, ICLR.

[4]  Federico Tombari,et al.  3D Point Capsule Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Nitish Srivastava,et al.  Capsules with Inverted Dot-Product Attention Routing , 2020, ICLR.

[6]  Anselm Haselhoff,et al.  Do Capsule Networks Solve the Problem of Rotation Invariance for Traffic Sign Classification? , 2018, ICANN.

[7]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[8]  Yee Whye Teh,et al.  Stacked Capsule Autoencoders , 2019, NeurIPS.

[9]  Hien Van Nguyen,et al.  Fast CapsNet for Lung Cancer Screening , 2018, MICCAI.

[10]  Lihui Chen,et al.  Capsule Graph Neural Network , 2018, ICLR.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  Ashley Daniel Gritzman.  Avoiding Implementation Pitfalls of "Matrix Capsules with EM Routing" by Hinton et al , 2019, HBAI@IJCAI.

[13]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[14]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[15]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[16]  Gideon Kowadlo,et al.  Sparse Unsupervised Capsules Generalize Better , 2018, ArXiv.

[17]  Y. LeCun,et al.  Learning methods for generic object recognition with invariance to pose and lighting , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[18]  Dai Quoc Nguyen,et al.  A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization , 2018, NAACL.