Attentive multi-view deep subspace clustering net

In this paper, we propose a novel Attentive Multi-View Deep Subspace Nets (AMVDSN), which deeply explores underlying consistent and view-specific information from multiple views and fuse them by considering each view’s dynamic contribution obtained by attention mechanism. Unlike most multi-view subspace learning methods that they directly reconstruct data points on raw data or only consider consistency or complementarity when learning representation in deep or shallow space, our proposed method seeks to find a joint latent representation that explicitly considers both consensus and view-specific information among multiple views, and then performs subspace clustering on learned joint latent representation.Besides, different views contribute differently to representation learning, we therefore introduce attention mechanism to derive dynamic weight for each view, which performs much better than previous fusion methods in the field of multi-view subspace clustering. The proposed algorithm is intuitive and can be easily optimized just by using Stochastic Gradient Descent (SGD) because of the neural network framework, which also provides strong non-linear characterization capability compared with traditional subspace clustering approaches. The experimental results on seven real-world data sets have demonstrated the effectiveness of our proposed algorithm against some state-of-the-art subspace learning approaches.

[1]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[2]  René Vidal,et al.  Kernel sparse subspace clustering , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[3]  Yong Luo,et al.  Tensor Canonical Correlation Analysis for Multi-View Dimension Reduction , 2015, IEEE Trans. Knowl. Data Eng..

[4]  Xianchao Zhang,et al.  Multi-Task Multi-View Clustering , 2016, IEEE Transactions on Knowledge and Data Engineering.

[5]  Ion Muslea,et al.  Active Learning with Multiple Views , 2009, Encyclopedia of Data Warehousing and Mining.

[6]  Dinggang Shen,et al.  Infant brain development prediction with latent partial multi-view representation learning , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[7]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[8]  Shiliang Sun,et al.  Multiple-view multiple-learner active learning , 2010, Pattern Recognit..

[9]  Xin Zuo,et al.  Multi-view representation learning in multi-task scene , 2019, Neural Computing and Applications.

[10]  Tomislav Šmuc,et al.  The landscape of microbial phenotypic traits and associated genes , 2016, Nucleic acids research.

[11]  Qinghua Hu,et al.  Generalized Latent Multi-View Subspace Clustering , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[13]  Hong Peng,et al.  Enhancing multi-view clustering through common subspace integration by considering both global similarities and local structures , 2020, Neurocomputing.

[14]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[15]  Vishal M. Patel,et al.  Deep Multimodal Subspace Clustering Networks , 2018, IEEE Journal of Selected Topics in Signal Processing.

[16]  René Vidal,et al.  Latent Space Sparse Subspace Clustering , 2013, 2013 IEEE International Conference on Computer Vision.

[17]  Yong Shi,et al.  Laplacian twin support vector machine for semi-supervised classification , 2012, Neural Networks.

[18]  Changqing Zhang,et al.  Multi-view Deep Subspace Clustering Networks , 2019, ArXiv.

[19]  Zenglin Xu,et al.  Auto-weighted multi-view clustering via kernelized graph learning , 2019, Pattern Recognit..

[20]  Xiaochun Cao,et al.  Diversity-induced Multi-view Subspace Clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Qiang Yang,et al.  Multi-transfer: Transfer learning with multiple views and multiple sources , 2014 .

[22]  René Vidal,et al.  Sparse Subspace Clustering: Algorithm, Theory, and Applications , 2012, IEEE transactions on pattern analysis and machine intelligence.

[23]  Jun Wang,et al.  Feature Concatenation Multi-view Subspace Clustering , 2019, Neurocomputing.

[24]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[25]  Shiliang Sun,et al.  View Construction for Multi-view Semi-supervised Learning , 2011, ISNN.

[26]  Qiang Cheng,et al.  Discriminative Ridge Machine: A Classifier for High-Dimensional Data or Imbalanced Data. , 2020, IEEE transactions on neural networks and learning systems.

[27]  Tao Mei,et al.  Subspace Clustering by Block Diagonal Representation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Yann LeCun,et al.  Effiicient BackProp , 1996, Neural Networks: Tricks of the Trade.

[29]  Feiping Nie,et al.  Multi-Subspace Representation and Discovery , 2011, ECML/PKDD.

[30]  Shiliang Sun,et al.  Multi-view Regularized Gaussian Processes , 2017, PAKDD.

[31]  Feiping Nie,et al.  Detecting Coherent Groups in Crowd Scenes by Multiview Clustering , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Hongdong Li,et al.  Efficient dense subspace clustering , 2014, IEEE Winter Conference on Applications of Computer Vision.

[33]  Qinghua Hu,et al.  Latent Multi-view Subspace Clustering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[35]  Evan Wei Xiang,et al.  Multi‐transfer: Transfer learning with multiple views and multiple sources , 2014, Stat. Anal. Data Min..

[36]  Sepp Hochreiter,et al.  Self-Normalizing Neural Networks , 2017, NIPS.

[37]  Shuicheng Yan,et al.  Robust Subspace Segmentation with Block-Diagonal Prior , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Junbin Gao,et al.  Kernel Sparse Subspace Clustering on Symmetric Positive Definite Manifolds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Hongdong Li,et al.  Shape Interaction Matrix Revisited and Robustified: Efficient Subspace Clustering with Corrupted and Incomplete Data , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[40]  Dong Xu,et al.  Robust Kernel Low-Rank Representation , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[41]  Dacheng Tao,et al.  Multi-View Intact Space Learning , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Jing Liu,et al.  Partially Shared Latent Factor Learning With Multiview Data , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[43]  Tao Zhou,et al.  Dual Shared-Specific Multiview Subspace Clustering , 2020, IEEE Transactions on Cybernetics.

[44]  Ivica Kopriva,et al.  Multi-view low-rank sparse subspace clustering , 2017, Pattern Recognit..

[45]  Wei-Yun Yau,et al.  Structured AutoEncoders for Subspace Clustering , 2018, IEEE Transactions on Image Processing.

[46]  Shuicheng Yan,et al.  Robust and Efficient Subspace Segmentation via Least Squares Regression , 2012, ECCV.

[47]  Wei Zhang,et al.  Consistent and Specific Multi-View Subspace Clustering , 2018, AAAI.

[48]  Tong Zhang,et al.  Deep Subspace Clustering Networks , 2017, NIPS.

[49]  Yiming Yang,et al.  RCV1: A New Benchmark Collection for Text Categorization Research , 2004, J. Mach. Learn. Res..

[50]  Jiancheng Lv,et al.  COMIC: Multi-view Clustering Without Parameter Selection , 2019, ICML.

[51]  Jingjing Tang,et al.  Multiview Privileged Support Vector Machines , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[52]  Xiaochun Cao,et al.  Low-Rank Tensor Constrained Multiview Subspace Clustering , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[53]  René Vidal,et al.  Low rank subspace clustering (LRSC) , 2014, Pattern Recognit. Lett..

[54]  Chuhan Wu,et al.  Neural Demographic Prediction using Search Query , 2019, WSDM.

[55]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  René Vidal,et al.  Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Shiliang Sun,et al.  Semi-supervised multi-view maximum entropy discrimination with expectation Laplacian regularization , 2019, Inf. Fusion.

[58]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[59]  Xiaodong He,et al.  A Multi-View Deep Learning Approach for Cross Domain User Modeling in Recommendation Systems , 2015, WWW.

[60]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[61]  Hao Wang,et al.  Multi-view clustering: A survey , 2018, Big Data Min. Anal..

[62]  Hai Liu,et al.  A Multi-View Clustering Method for Community Discovery Integrating Links and Tags , 2017, 2017 IEEE 14th International Conference on e-Business Engineering (ICEBE).