Semi-supervised Clustering using Combinatorial MRFs

A combinatorial random variable is a discrete random variable defined over a combinatorial set (e.g., a power set of a given set). In this paper we introduce combinatorial Markov random fields (Comrafs), which are Markov random fields where some of the nodes are combinatorial random variables. We argue that Comrafs are powerful models for unsupervised learning by showing their relationship with two existing models. We then present a Comraf model for semi-supervised clustering that demonstrates superior results in comparison to an existing semi-supervised scheme (constrained optimization).

[1]  William T. Freeman,et al.  Learning low-level vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[3]  Claire Cardie,et al.  Clustering with Instance-Level Constraints , 2000, AAAI/IAAI.

[4]  W. Bruce Croft,et al.  A Markov random field model for term dependencies , 2005, SIGIR '05.

[5]  Erik G. Learned-Miller,et al.  Combinatorial Markov Random Fields , 2006, ECML.

[6]  Fernando Pereira,et al.  Shallow Parsing with Conditional Random Fields , 2003, NAACL.

[7]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.

[8]  Stan Z. Li,et al.  Markov Random Field Modeling in Computer Vision , 1995, Computer Science Workbench.

[9]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[10]  J. Besag Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[11]  Raymond J. Mooney,et al.  A probabilistic framework for semi-supervised clustering , 2004, KDD.

[12]  Michael I. Jordan Graphical Models , 2003 .

[13]  Inderjit S. Dhillon,et al.  Information-theoretic co-clustering , 2003, KDD '03.

[14]  Naftali Tishby,et al.  Multivariate Information Bottleneck , 2001, Neural Computation.

[15]  J. Besag On the Statistical Analysis of Dirty Pictures , 1986 .

[16]  Ran El-Yaniv,et al.  Distributional Word Clusters vs. Words for Text Categorization , 2003, J. Mach. Learn. Res..

[17]  Ran El-Yaniv,et al.  Multi-way distributional clustering via pairwise interactions , 2005, ICML.