Self-organizing maps: Generalizations and new optimization techniques

Abstract We offer three algorithms for the generation of topographic mappings to the practitioner of unsupervised data analysis. The algorithms are each based on the minimization of a cost function which is performed using an EM algorithm and deterministic annealing. The soft topographic vector quantization algorithm (STVQ) – like the original self-organizing map (SOM) – provides a tool for the creation of self-organizing maps of Euclidean data. Its optimization scheme, however, offers an alternative to the heuristic stepwise shrinking of the neighborhood width in the SOM and makes it possible to use a fixed neighborhood function solely to encode desired neighborhood relations between nodes. The kernel-based soft topographic mapping (STMK) is a generalization of STVQ and introduces new distance measures in data space, based on kernel functions. Using the new distance measures corresponds to performing the STVQ in a high-dimensional feature space, which is related to data space by a nonlinear mapping. This preprocessing can reveal structure in the data which may go unnoticed if the STVQ is performed in the standard Euclidean space. The soft topographic mapping for proximity data (STMP) is another generalization of STVQ that enables the user to generate topographic maps for data which are given in terms of pairwise proximities. It thus offers a flexible alternative to multidimensional scaling methods and opens up a new range of applications for SOMs. Both STMK and STMP share the robust optimization properties of STVQ due to the application of deterministic annealing. In our contribution we discuss the algorithms together with their implementation and provide detailed pseudo-code and explanations.

[1]  S. P. Luttrell,et al.  A Bayesian Analysis of Self-Organizing Maps , 1994, Neural Computation.

[2]  Rose,et al.  Statistical mechanics and phase transitions in clustering. , 1990, Physical review letters.

[3]  Jorma Laaksonen,et al.  SOM_PAK: The Self-Organizing Map Program Package , 1996 .

[4]  Klaus Obermayer,et al.  An Annealed Self-Organizing Map for Source Channel Coding , 1997, NIPS.

[5]  T. Kohonen,et al.  Self-Organizing Maps of Symbol Strings with Application to Speech Recognition , 1997 .

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[7]  Eric Mjolsness,et al.  Learning with Preknowledge: Clustering with Point and Graph Matching Distance Measures , 1996, Neural Computation.

[8]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[9]  S. Lutrell.  Code Vector Density in Topographic Mappings , 1992 .

[10]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[11]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[12]  Joachim M. Buhmann,et al.  An optimization approach to unsupervised hierarchical texture segmentation , 1997, Proceedings of International Conference on Image Processing.

[13]  Kenneth Rose,et al.  Combined source-channel vector quantization using deterministic annealing , 1994, IEEE Trans. Commun..

[14]  Stephen P. Luttrell.  Code vector density in topographic mappings: Scalar case , 1991, IEEE Trans. Neural Networks.

[15]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[16]  Bernhard Schölkopf,et al.  Extracting Support Data for a Given Task , 1995, KDD.

[17]  N. Swindale.  The development of topography in the visual cortex: a review of models. , 1996, Network.

[18]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[19]  Klaus Obermayer,et al.  A Stochastic Self-Organizing Map for Proximity Data , 1999, Neural Computation.

[20]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.

[21]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[22]  Klaus Schulten,et al.  Models of Orientation and Ocular Dominance Columns in the Visual Cortex: A Critical Comparison , 1995, Neural Computation.

[23]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[24]  K. Obermayer,et al.  PHASE TRANSITIONS IN STOCHASTIC SELF-ORGANIZING MAPS , 1997 .

[25]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[26]  J. J. Kosowsky,et al.  Statistical Physics Algorithms That Converge , 1994, Neural Computation.

[27]  Joachim M. Buhmann,et al.  Vector quantization with complexity costs , 1993, IEEE Trans. Inf. Theory.

[28]  Helge Ritter,et al.  Learning with the Self-Organizing Map , 1991 .

[29]  Klaus Obermayer,et al.  Phase Transitions in Soft Topographic Vector Quantization , 1997, ICANN.

[30]  Hilbert J. Kappen,et al.  Error potentials for self-organization , 1993, IEEE International Conference on Neural Networks.

[31]  Michael I. Jordan,et al.  Exploiting Tractable Substructures in Intractable Networks , 1995, NIPS.

[32]  GraepelMatthias,et al.  Deterministic Annealing for Topographic VectorQuantization and Self-Organizing MapsThore , 1997 .