Cooperative information maximization with Gaussian activation functions for self-organizing maps

In this paper, we propose a new information-theoretic method to produce explicit self-organizing maps (SOMs). Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the Gaussian function of distance between input patterns and competitive units. A property of this Gaussian function is that, as distance becomes smaller, a neuron tends to fire strongly. Cooperation processes are realized by taking into account the firing rates of neighboring neurons. We applied our method to uniform distribution learning, chemical compound classification and road classification. Experimental results confirmed that cooperation processes could significantly increase information content in input patterns. When cooperative operations are not effective in increasing information, mutual information as well as entropy maximization is used to increase information. Experimental results showed that entropy maximization could be used to increase information and to obtain clearer SOMs, because competitive units are forced to be equally used on average.

[1]  Jiun-In Guo,et al.  A new k-winners-take-all neural network and its array architecture , 1998, IEEE Trans. Neural Networks.

[2]  Tom Heskes,et al.  Self-organizing maps, vector quantization, and mixture modeling , 2001, IEEE Trans. Neural Networks.

[3]  Claude E. Shannon,et al.  A mathematical theory of communication , 1948, MOCO.

[4]  J. H. Ward.  Hierarchical Grouping to Optimize an Objective Function , 1963 .

[5]  James L. McClelland,et al.  On learning the past-tenses of English verbs: implicit rules or parallel distributed processing , 1986 .

[6]  Ralph Linsker,et al.  Self-organization in a perceptual network , 1988, Computer.

[7]  Duane DeSieno,et al.  Adding a conscience to competitive learning , 1988, IEEE 1988 International Conference on Neural Networks.

[8]  Corneliu A. Marinov,et al.  Another K-winners-take-all analog neural network , 2000, IEEE Trans. Neural Networks Learn. Syst..

[9]  Ralph Linsker,et al.  How to Generate Ordered Maps by Maximizing the Mutual Information between Input and Output Signals , 1989, Neural Computation.

[10]  J. Kruskal.  Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis , 1964 .

[11]  Marc M. Van Hulle,et al.  Faithful representations with topographic maps , 1999, Neural Networks.

[12]  Marc M. Van Hulle.  Topographic map formation by maximizing unconditional entropy: a plausible strategy for "online" unsupervised competitive learning and nonparametric density estimation , 1996, IEEE Trans. Neural Networks.

[13]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[14]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[15]  Erkki Oja,et al.  Rival penalized competitive learning for clustering analysis, RBF net, and curve detection , 1993, IEEE Trans. Neural Networks.

[16]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[17]  Ralph Linsker,et al.  Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network , 1992, Neural Computation.

[18]  Marc M. Van Hulle,et al.  The Formation of Topographic Maps That Maximize the Average Mutual Information of the Output Responses to Noiseless Input Signals , 1997, Neural Computation.

[19]  Ryotaro Kamimura,et al.  Information Theoretic Competitive Learning and Linguistic Rule Acquisition , 2001 .

[20]  Marc M. Van Hulle,et al.  Entropy-based kernel mixture modeling for topographic map formation , 2004, IEEE Transactions on Neural Networks.

[21]  Ryotaro Kamimura,et al.  Flexible feature discovery and structural information control , 2001, Connect. Sci..

[22]  A. Luk,et al.  General properties of the generalised Lotto-type competitive learning , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[23]  Stanley C. Ahalt,et al.  Competitive learning algorithms for vector quantization , 1990, Neural Networks.

[24]  Marc M. Van Hulle,et al.  Topology-preserving Map Formation Achieved with a Purely Local Unsupervised Competitive Learning Rule , 1997, Neural Networks.

[25]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[26]  Ryotaro Kamimura,et al.  Information-Theoretic Competitive Learning with Inverse Euclidean Distance Output Units , 2003, Neural Processing Letters.