Model-Based Clustering by Probabilistic Self-Organizing Maps

In this paper, we consider the learning process of a probabilistic self-organizing map (PbSOM) as a model-based data clustering procedure that preserves the topological relationships between data clusters in a neural network. Based on this concept, we develop a coupling-likelihood mixture model for the PbSOM that extends the reference vectors in Kohonen's self-organizing map (SOM) to multivariate Gaussian distributions. We also derive three expectation-maximization (EM)-type algorithms, called the SOCEM, SOEM, and SODAEM algorithms, for learning the model (PbSOM) based on the maximum-likelihood criterion. SOCEM is derived by using the classification EM (CEM) algorithm to maximize the classification likelihood; SOEM is derived by using the EM algorithm to maximize the mixture likelihood; and SODAEM is a deterministic annealing (DA) variant of SOCEM and SOEM. Moreover, by shrinking the neighborhood size, SOCEM and SOEM can be interpreted, respectively, as DA variants of the CEM and EM algorithms for Gaussian model-based clustering. The experimental results show that the proposed PbSOM learning algorithms achieve comparable data clustering performance to that of the deterministic annealing EM (DAEM) approach, while maintaining the topology-preserving property.

[1]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.

[2]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[3]  Fouad Badran,et al.  Probabilistic self-organizing map and radial basis function networks , 1998, Neurocomputing.

[4]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[5]  G. Govaert,et al.  Constrained clustering and Kohonen Self-Organizing Maps , 1996 .

[6]  Marc M. Van Hulle.  Joint Entropy Maximization in Kernel-Based Topographic Maps , 2002, Neural Computation.

[7]  Stephen P. Luttrell.  Code vector density in topographic mappings: Scalar case , 1991, IEEE Trans. Neural Networks.

[8]  V. V. Tolat.  An analysis of Kohonen's self-organizing maps using a system of energy functions , 1990, Biological Cybernetics.

[9]  Adrian E. Raftery,et al.  Bayesian Regularization for Normal Mixture Estimation and Model-Based Clustering , 2007, J. Classif..

[10]  Michael J. Symons,et al.  Clustering criteria and multivariate normal mixtures , 1981 .

[11]  Jeff A. Bilmes,et al.  A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models , 1998 .

[12]  Tommy W. S. Chow,et al.  An online cellular probabilistic self-organizing map for static and dynamic data sets , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[13]  Klaus Obermayer,et al.  Self-organizing maps: Generalizations and new optimization techniques , 1998, Neurocomputing.

[14]  Klaus Schulten,et al.  Self-organizing maps: ordering, convergence properties and energy functions , 1992, Biological Cybernetics.

[15]  G. Celeux,et al.  A Classification EM algorithm for clustering and two stochastic versions , 1992 .

[16]  S. P. Luttrell,et al.  Self-organisation: a derivation from first principles of a class of learning algorithms , 1989, International 1989 Joint Conference on Neural Networks.

[17]  Marc M. Van Hulle,et al.  Maximum Likelihood Topographic Map Formation , 2005, Neural Computation.

[18]  Tom Heskes,et al.  Self-organizing maps, vector quantization, and mixture modeling , 2001, IEEE Trans. Neural Networks.

[19]  A. Raftery,et al.  Model-Based Clustering With Dissimilarities: A Bayesian Approach , 2007 .

[20]  Geoffrey C. Fox,et al.  Vector quantization by deterministic annealing , 1992, IEEE Trans. Inf. Theory.

[21]  Tommy W. S. Chow,et al.  PRSOM: a new visualization method by hybridizing multidimensional scaling and self-organizing map , 2005, IEEE Transactions on Neural Networks.

[22]  Geoffrey E. Hinton,et al.  SMEM Algorithm for Mixture Models , 1998, Neural Computation.

[23]  K. Obermayer,et al.  PHASE TRANSITIONS IN STOCHASTIC SELF-ORGANIZING MAPS , 1997 .

[24]  Timo Kostiainen,et al.  Generative probability density model in the self-organizing map , 2001 .

[25]  A. Raftery,et al.  Model-based Gaussian and non-Gaussian clustering , 1993 .

[26]  Adrian E. Raftery,et al.  How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis , 1998, Comput. J..

[27]  Hsin-Min Wang,et al.  A Model-Selection-Based Self-Splitting Gaussian Mixture Learning with Application to Speaker Identification , 2004, EURASIP J. Adv. Signal Process..

[28]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[29]  Andrew Chi-Sing Leung,et al.  Yet another algorithm which can generate topography map , 1997, IEEE Trans. Neural Networks.

[30]  S. P. Luttrell,et al.  A Bayesian Analysis of Self-Organizing Maps , 1994, Neural Computation.

[31]  S. Angel Latha Mary,et al.  Classification and Mixture Approaches to Clustering via Maximum Likelihood , 1989 .

[32]  Jouko Lampinen,et al.  On the generative probability density model in the self-organizing map , 2002, Neurocomputing.

[33]  Yizong Cheng.  Convergence and Ordering of Kohonen's Batch Map , 1997, Neural Computation.

[34]  M. V. Velzen,et al.  Self-organizing maps , 2007 .

[35]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[36]  Ben J. A. Kröse,et al.  Self-organizing mixture models , 2005, Neurocomputing.

[37]  Joydeep Ghosh,et al.  A Unified Framework for Model-based Clustering , 2003, J. Mach. Learn. Res..

[38]  Naonori Ueda,et al.  Deterministic annealing EM algorithm , 1998, Neural Networks.

[39]  K. Rose.  Deterministic annealing for clustering, compression, classification, regression, and related optimization problems , 1998, Proc. IEEE.