DIVA: A Dirichlet Process Based Incremental Deep Clustering Algorithm via Variational Auto-Encoder

Generative model-based deep clustering frameworks excel in classifying complex data, but are limited in handling dynamic and complex features because they require prior knowledge of the number of clusters. In this paper, we propose a nonparametric deep clustering framework that employs an infinite mixture of Gaussians as a prior. Our framework utilizes a memoized online variational inference method that enables the"birth"and"merge"moves of clusters, allowing our framework to cluster data in a"dynamic-adaptive"manner, without requiring prior knowledge of the number of features. We name the framework as DIVA, a Dirichlet Process-based Incremental deep clustering framework via Variational Auto-Encoder. Our framework, which outperforms state-of-the-art baselines, exhibits superior performance in classifying complex data with dynamically changing features, particularly in the case of incremental features. We released our source code implementation at: https://github.com/Ghiara/diva

[1]  F. O. Morin,et al.  Meta-Reinforcement Learning in Nonstationary and Nonparametric Environments. , 2023, IEEE transactions on neural networks and learning systems.

[2]  Hang Su,et al.  Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning , 2023, AAAI.

[3]  Kai Huang,et al.  Meta-Reinforcement Learning in Non-Stationary and Dynamic Environments , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jiawei Chen,et al.  A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions , 2022, ArXiv.

[5]  Tianjia Shao,et al.  Unsupervised Image Generation with Infinite Generative Adversarial Networks , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Hyunwoo J. Kim,et al.  Online Continual Learning in Image Classification: An Empirical Survey , 2021, Neurocomputing.

[7]  Yaochu Jin,et al.  Image Clustering Using an Augmented Generative Adversarial Network and Information Maximization , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Luc Van Gool,et al.  SCAN: Learning to Classify Images Without Labels , 2020, ECCV.

[9]  Daniel K Hartline,et al.  t-Distributed Stochastic Neighbor Embedding (t-SNE): A tool for eco-physiological transcriptomic analysis. , 2019, Marine genomics.

[10]  Tinne Tuytelaars,et al.  A Continual Learning Survey: Defying Forgetting in Classification Tasks , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Mithat Gönen,et al.  A tutorial on Dirichlet Process mixture modeling. , 2019, Journal of mathematical psychology.

[12]  Yang Yu,et al.  Mixture of GANs for Clustering , 2018, IJCAI.

[13]  Xindong Wu,et al.  Short text clustering based on Pitman-Yor process mixture model , 2018, Applied Intelligence.

[14]  Shuigeng Zhou,et al.  DeepCluster: A General Clustering Framework Based on Deep Learning , 2017, ECML/PKDD.

[15]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[16]  Zeb Kurth-Nelson,et al.  Learning to reinforcement learn , 2016, CogSci.

[17]  Huachun Tan,et al.  Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering , 2016, IJCAI.

[18]  Murray Shanahan,et al.  Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders , 2016, ArXiv.

[19]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[20]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[21]  Jianyong Wang,et al.  A dirichlet multinomial mixture model-based approach for short text clustering , 2014, KDD.

[22]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[23]  Erik B. Sudderth,et al.  Memoized Online Variational Inference for Dirichlet Process Mixture Models , 2013, NIPS.

[24]  Oliver Kramer,et al.  Dimensionality Reduction with Unsupervised Nearest Neighbors , 2013, Intelligent Systems Reference Library.

[25]  Charu C. Aggarwal,et al.  A Survey of Text Clustering Algorithms , 2012, Mining Text Data.

[26]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[27]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[28]  Yee Whye Teh,et al.  Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes , 2004, NIPS.

[29]  Thomas L. Griffiths,et al.  Hierarchical Topic Models and the Nested Chinese Restaurant Process , 2003, NIPS.

[30]  Anthony V. Robins,et al.  Catastrophic Forgetting, Rehearsal and Pseudorehearsal , 1995, Connect. Sci..

[31]  Kisung Seo,et al.  GAN-Based Anomaly Detection In Imbalance Problems , 2020, ECCV Workshops.

[32]  Marco Cote.  STICK-BREAKING VARIATIONAL AUTOENCODERS , 2017 .