Model-Based Clustering by Probabilistic

In this paper, we consider the learning process of a probabilistic self-organizing map (PbSOM) as a model-based data clustering procedure that preserves the topological relationships between data clusters in a neural network. Based on this concept, we develop a coupling-likelihood mixture model for the PbSOM that extends the reference vectors in Kohonen's self-organizing map (SOM) to multivariate Gaussian distributions. We also derive three expectation-maximization (EM)-type algorithms, called the SOCEM, SOEM, and SODAEM algorithms, for learning the model (PbSOM) based on the maximum-likelihood crite- rion. SOCEM is derived by using the classification EM (CEM) algorithm to maximize the classification likelihood; SOEM is derived by using the EM algorithm to maximize the mixture like- lihood; and SODAEM is a deterministic annealing (DA) variant of SOCEM and SOEM. Moreover, by shrinking the neighbor- hood size, SOCEM and SOEM can be interpreted, respectively, as DA variants of the CEM and EM algorithms for Gaussian model-based clustering. The experimental results show that the proposed PbSOM learning algorithms achieve comparable data clustering performance to that of the deterministic annealing EM (DAEM) approach, while maintaining the topology-preserving property. Index Terms—Classification expectation-maximization (CEM) algorithm, deterministic annealing expectation-maximization (DAEM) algorithm, expectation-maximization (EM) algo- rithm, model-based clustering, probabilistic self-organizing map (PbSOM), self-organizing map (SOM).

[1]  G. Celeux,et al.  A Classification EM algorithm for clustering and two stochastic versions , 1992 .

[2]  Geoffrey C. Fox,et al.  Vector quantization by deterministic annealing , 1992, IEEE Trans. Inf. Theory.

[3]  Fouad Badran,et al.  Probabilistic self-organizing map and radial basis function networks , 1998, Neurocomputing.

[4]  A. Raftery,et al.  Model-based Gaussian and non-Gaussian clustering , 1993 .

[5]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[6]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[7]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[8]  Timo Kostiainen,et al.  Generative probability density model in the self-organizing map , 2001 .

[9]  K. Rose.  Deterministic annealing for clustering, compression, classification, regression, and related optimization problems , 1998, Proc. IEEE.

[10]  G. Govaert,et al.  Constrained clustering and Kohonen Self-Organizing Maps , 1996 .

[11]  Michael J. Symons,et al.  Clustering criteria and multivariate normal mixtures , 1981 .

[12]  Marc M. Van Hulle.  Joint Entropy Maximization in Kernel-Based Topographic Maps , 2002, Neural Computation.

[13]  Stephen P. Luttrell.  Code vector density in topographic mappings: Scalar case , 1991, IEEE Trans. Neural Networks.

[14]  Klaus Obermayer,et al.  Self-organizing maps: Generalizations and new optimization techniques , 1998, Neurocomputing.

[15]  Klaus Schulten,et al.  Self-organizing maps: ordering, convergence properties and energy functions , 1992, Biological Cybernetics.

[16]  Tommy W. S. Chow,et al.  PRSOM: a new visualization method by hybridizing multidimensional scaling and self-organizing map , 2005, IEEE Transactions on Neural Networks.

[17]  Yizong Cheng.  Convergence and Ordering of Kohonen's Batch Map , 1997, Neural Computation.

[18]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[19]  Adrian E. Raftery,et al.  How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis , 1998, Comput. J..

[20]  Hsin-Min Wang,et al.  A Model-Selection-Based Self-Splitting Gaussian Mixture Learning with Application to Speaker Identification , 2004, EURASIP J. Adv. Signal Process..

[21]  S. P. Luttrell,et al.  A Bayesian Analysis of Self-Organizing Maps , 1994, Neural Computation.

[22]  Geoffrey E. Hinton,et al.  SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[23]  K. Obermayer,et al.  PHASE TRANSITIONS IN STOCHASTIC SELF-ORGANIZING MAPS , 1997 .

[24]  V. V. Tolat.  An analysis of Kohonen's self-organizing maps using a system of energy functions , 1990, Biological Cybernetics.

[25]  Adrian E. Raftery,et al.  Bayesian Regularization for Normal Mixture Estimation and Model-Based Clustering , 2007, J. Classif..

[26]  Andrew Chi-Sing Leung,et al.  Yet another algorithm which can generate topography map , 1997, IEEE Trans. Neural Networks.

[27]  Jeff A. Bilmes,et al.  A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models , 1998 .

[28]  Marc M. Van Hulle,et al.  Maximum Likelihood Topographic Map Formation , 2005, Neural Computation.

[29]  Tom Heskes,et al.  Self-organizing maps, vector quantization, and mixture modeling , 2001, IEEE Trans. Neural Networks.

[30]  A. Raftery,et al.  Model-Based Clustering With Dissimilarities: A Bayesian Approach , 2007 .

[31]  S. Ganesalingam.  Classification and Mixture Approaches to Clustering Via Maximum Likelihood , 1989 .

[32]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[33]  Naonori Ueda,et al.  Deterministic annealing EM algorithm , 1998, Neural Networks.

[34]  Ben J. A. Kröse,et al.  Self-organizing mixture models , 2005, Neurocomputing.

[35]  Joydeep Ghosh,et al.  A Unified Framework for Model-based Clustering , 2003, J. Mach. Learn. Res..

[36]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .