Deep Discriminative Clustering Analysis

Traditional clustering methods often perform clustering with low-level indiscriminative representations and ignore relationships between patterns, resulting in slight achievements in the era of deep learning. To handle this problem, we develop Deep Discriminative Clustering (DDC) that models the clustering task by investigating relationships between patterns with a deep neural network. Technically, a global constraint is introduced to adaptively estimate the relationships, and a local constraint is developed to endow the network with the capability of learning high-level discriminative representations. By iteratively training the network and estimating the relationships in a mini-batch manner, DDC theoretically converges and the trained network enables to generate a group of discriminative representations that can be treated as clustering centers for straightway clustering. Extensive experiments strongly demonstrate that DDC outperforms current methods on eight image, text and audio datasets concurrently.

[1]  Sreeram Kannan,et al.  ClusterGAN : Latent Space Clustering in Generative Adversarial Networks , 2018, AAAI.

[2]  Feiping Nie,et al.  Discriminatively Embedded K-Means for Multi-view Clustering , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Vladimir Pavlovic,et al.  Probabilistic Temporal Subspace Clustering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Zhuowen Tu,et al.  Probabilistic boosting-tree: learning discriminative models for classification, recognition, and clustering , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[5]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[6]  Pasi Fränti,et al.  Fast Agglomerative Clustering Using a k-Nearest Neighbor Graph , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Andreas Krause,et al.  Discriminative Clustering by Regularized Information Maximization , 2010, NIPS.

[8]  René Vidal,et al.  Structured Sparse Subspace Clustering: A unified optimization framework , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Erkki Oja,et al.  Low-Rank Doubly Stochastic Matrix Decomposition for Cluster Analysis , 2016, J. Mach. Learn. Res..

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[12]  Yann LeCun,et al.  Stacked What-Where Auto-encoders , 2015, ArXiv.

[13]  Sabine Süsstrunk,et al.  Superpixels and Polygons Using Simple Non-iterative Clustering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Christos V. Verikoukis,et al.  Self-Tuning Spectral Clustering for Adaptive Tracking Areas Design in 5G Ultra-Dense Networks , 2019, 2019 IEEE Wireless Communications and Networking Conference (WCNC).

[15]  Heng Tao Shen,et al.  Optimized Cartesian K-Means , 2014, IEEE Transactions on Knowledge and Data Engineering.

[16]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Pierre Vandergheynst,et al.  Fast Approximate Spectral Clustering for Dynamic Networks , 2017, ICML.

[18]  Aren Jansen,et al.  Audio Set: An ontology and human-labeled dataset for audio events , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Frédéric Jurie,et al.  Fast Discriminative Visual Codebooks using Randomized Clustering Forests , 2006, NIPS.

[20]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[21]  Chris H. Q. Ding,et al.  Discriminative high order SVD: Adaptive tensor subspace selection for image classification, clustering, and retrieval , 2011, 2011 International Conference on Computer Vision.

[22]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[23]  Gunnar Rätsch,et al.  Clustering Meets Implicit Generative Models , 2018, ICLR.

[24]  Zhao Kang,et al.  Subspace Clustering via Variance Regularized Ridge Regression , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Huachun Tan,et al.  Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering , 2016, IJCAI.

[27]  Chao Chen,et al.  Clustering High Dimensional Categorical Data via Topographical Features , 2016, ICML.

[28]  Loïc Le Folgoc,et al.  Semi-Supervised Learning via Compact Latent Space Clustering , 2018, ICML.

[29]  Ismail Ben Ayed,et al.  Kernel Clustering: Density Biases and Solutions , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[32]  Feiping Nie,et al.  A Self-Balanced Min-Cut Algorithm for Image Clustering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Jie Gu,et al.  Structure-Aware Convolutional Neural Networks , 2018, NeurIPS.

[34]  Aditya Bhaskara,et al.  Distributed Clustering via LSH Based Data Partitioning , 2018, ICML.

[35]  Cheng Deng,et al.  Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Jiashi Feng,et al.  Deep Adversarial Subspace Clustering , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Bjoern Andres,et al.  Partial Optimality and Fast Lower Bounds for Weighted Correlation Clustering , 2018, ICML.

[38]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[39]  Kiyoharu Aizawa,et al.  Local and Global Optimization Techniques in Graph-Based Clustering , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Alan L. Yuille,et al.  Unsupervised Learning Using Generative Adversarial Training And Clustering , 2016 .

[41]  Rudolf Mester,et al.  Discriminative Subspace Clustering , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[43]  Weisi Lin,et al.  Learning Markov Clustering Networks for Scene Text Detection , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[44]  Francesc Moreno-Noguer,et al.  Image Collection Pop-up: 3D Reconstruction and Clustering of Rigid and Non-rigid Categories , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[46]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[47]  L. Hubert,et al.  Comparing partitions , 1985 .

[48]  Matthijs Douze,et al.  Deep Clustering for Unsupervised Learning of Visual Features , 2018, ECCV.

[49]  Yifan Sun,et al.  Deep Unsupervised Clustering Using Mixture of Autoencoders , 2017, ArXiv.

[50]  Murray Shanahan,et al.  Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders , 2016, ArXiv.

[51]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[52]  Junbin Gao,et al.  Kernel Sparse Subspace Clustering on Symmetric Positive Definite Manifolds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Yi-Leh Wu,et al.  Adaptive density-based spatial clustering of applications with noise (DBSCAN) according to data , 2015, 2015 International Conference on Machine Learning and Cybernetics (ICMLC).

[54]  Moses Charikar,et al.  Hierarchical Clustering with Structural Constraints , 2018, ICML.

[55]  Kaushik Sinha K-means clustering using random matrix sparsification , 2018, ICML.

[56]  Jian-Huang Lai,et al.  Dimensionality's Blessing: Clustering Images by Underlying Distribution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[57]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[58]  Wen-Sheng Chu,et al.  Learning Facial Action Units from Web Images with Scalable Weakly Supervised Clustering , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Takeo Kanade,et al.  Discriminative cluster analysis , 2006, ICML.

[60]  Gaofeng Meng,et al.  Deep Self-Evolution Clustering , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Tao Li,et al.  The Relationships Among Various Nonnegative Matrix Factorization Methods for Clustering , 2006, Sixth International Conference on Data Mining (ICDM'06).

[62]  Lingfeng Wang,et al.  Deep Adaptive Image Clustering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[63]  Ivan Laptev,et al.  Learning from Video and Text via Large-Scale Discriminative Clustering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[64]  Jieping Ye,et al.  Discriminative K-means for Clustering , 2007, NIPS.

[65]  Qinghua Hu,et al.  Latent Multiview Subspace Clustering , 2022 .

[66]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[67]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[68]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[69]  Yalin Wang,et al.  Variational Wasserstein Clustering , 2018, ECCV.

[70]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[71]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[72]  Daniel P. Robinson,et al.  Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Qi Tian,et al.  DisturbLabel: Regularizing CNN on the Loss Layer , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Jean Ponce,et al.  Discriminative clustering for image co-segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[75]  Sean Hughes,et al.  Clustering by Fast Search and Find of Density Peaks , 2016 .

[76]  Dhruv Batra,et al.  Joint Unsupervised Learning of Deep Representations and Image Clusters , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Mario Sznaier,et al.  SoS-RSC: A Sum-of-Squares Polynomial Approach to Robustifying Subspace Clustering Algorithms , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[78]  Junbin Gao,et al.  Grassmannian Manifold Optimization Assisted Sparse Spectral Clustering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[80]  Qi-Xing Huang,et al.  SMAC: Simultaneous Mapping and Clustering Using Spectral Decompositions , 2018, ICML.

[81]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[82]  Vladlen Koltun,et al.  Deep Continuous Clustering , 2018, ArXiv.

[83]  Lingfeng Wang,et al.  Deep unsupervised learning with consistent inference of latent representations , 2017, Pattern Recognit..

[84]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[85]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[86]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[87]  Yiming Yang,et al.  RCV1: A New Benchmark Collection for Text Categorization Research , 2004, J. Mach. Learn. Res..