Model-based Clustering With Soft And Probabilistic Constraints

The problem of clustering with constraints has received a lot of attention lately. Many existing algorithms assume the specified constraints are correct and consistent. We take a new approach and model a constraint as a random variable. This enables us to model the uncertainty of constraints in a principled manner. The effect of constraints can be readily propagated to the neighborhood by biasing the search of the optimal parameters in each cluster. This enforces “smooth” cluster labels. The posterior probabilities of these constraint random variables represent the a posteriori enforcement of the corresponding constraints. By combining these probability values with the data likelihood, we arrive at an objective function for parameter estimation. An EM algorithm that maximizes the lower bound of the objective function is derived for efficient parameter estimation, using the variational method. Experimental results demonstrate the usefulness of the proposed algorithm. In particular, our approach can identify the desired clusters when only a small portion of data participate in constraints.

[1]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[2]  Jianbo Shi,et al.  Segmentation given partial grouping constraints , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Dan Klein,et al.  From Instance-level Constraints to Space-Level Constraints: Making the Most of Prior Knowledge in Data Clustering , 2002, ICML.

[4]  Raymond J. Mooney,et al.  Integrating constraints and metric learning in semi-supervised clustering , 2004, ICML.

[5]  Tomer Hertz,et al.  Computing Gaussian Mixture Models with EM Using Equivalence Constraints , 2003, NIPS.

[6]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[7]  Anil K. Jain,et al.  Soft Biometric Traits for Personal Recognition Systems , 2004, ICBA.

[8]  Sebastian Thrun,et al.  Text Classification from Labeled and Unlabeled Documents using EM , 2000, Machine Learning.

[9]  Jianbo Shi,et al.  Grouping with Bias , 2001, NIPS.

[10]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[11]  Thomas Hofmann,et al.  Non-redundant data clustering , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[12]  Claire Cardie,et al.  Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584. Constrained K-means Clustering with Background Knowledge , 2022 .

[13]  Anil K. Jain,et al.  Clustering with Soft and Group Constraints , 2004, SSPR/SPR.

[14]  R. Tibshirani,et al.  Discriminant Analysis by Gaussian Mixtures , 1996 .

[15]  Alex Pentland,et al.  Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm , 1998, NIPS.

[16]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[17]  Raymond J. Mooney,et al.  A probabilistic framework for semi-supervised clustering , 2004, KDD.

[18]  Geoffrey E. Hinton,et al.  Global Coordination of Local Linear Models , 2001, NIPS.