LEARNING SELF-ORGANIZING MIXTURE MARKOV MODELS

This paper describes a new algorithm to learn SelfOrganizing map as Markov Mixture Models. Our model realizes an unsupervised learning using unlabelled evolutionary data sets, namely those that describe sequential data. The new formalism that we present is valid for all structure of graphical models. We use EM (Expectation-Maximisation) standard algorithm to maximize the likelihood. The graph structure is integrated in the parameter estimation of Markov model using a neighborhood function to learn a topographic clustering of not i.i.d data set. The new approach provides a self-organizing Markov model using an original learning algorithm. We provide three applications of our model: (1) dealing with continuous data using Gaussian distribution; (2) dealing with categorical data without any coding to encode variables using probability tables; (3) dealing with binary data using Bernoulli distribution.

[1]  L. Baum,et al.  An inequality and associated maximization technique in statistical estimation of probabilistic functions of a Markov process , 1972 .

[2]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[3]  Hervé Bourlard,et al.  Connectionist Speech Recognition: A Hybrid Approach , 1993 .

[4]  Yoshua Bengio,et al.  An Input Output HMM Architecture , 1994, NIPS.

[5]  S. P. Luttrell,et al.  A Bayesian Analysis of Self-Organizing Maps , 1994, Neural Computation.

[6]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[7]  Michael I. Jordan,et al.  Learning Fine Motion by Markov Mixtures of Experts , 1995, NIPS.

[8]  Samy Bengio,et al.  An EM Algorithm for Asynchronous Input/Output Hidden Markov Models , 1996 .

[9]  G. Govaert,et al.  Clustering for binary data and mixture models—choice of the model , 1997 .

[10]  Christopher M. Bishop,et al.  GTM: The Generative Topographic Mapping , 1998, Neural Computation.

[11]  Panu Somervuo.  Competing hidden Markov models on the self-organizing map , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[12]  Djamel Bouchaffra,et al.  Structural Hidden Markov Model and Its Application in Automotive Industry , 2003, ICEIS.

[13]  Younès Bennani,et al.  Hybrid connectionist approach for knowledge discovery from Web navigation patterns , 2003 .

[14]  Michael I. Jordan,et al.  Factorial Hidden Markov Models , 1995, Machine Learning.

[15]  Panu Somervuo,et al.  Self-Organizing Maps and Learning Vector Quantization for Feature Sequences , 1999, Neural Processing Letters.

[16]  Djamel Bouchaffra,et al.  Structural hidden Markov models: An application to handwritten numeral recognition , 2006, Intell. Data Anal..

[17]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[18]  Edoardo M. Airoldi,et al.  Getting Started in Probabilistic Graphical Models , 2007, PLoS Comput. Biol..

[19]  Younès Bennani,et al.  The structure of verbal sequences analyzed with unsupervised learning techniques , 2007, ArXiv.

[20]  Mustapha Lebbah,et al.  A Probabilistic Self-Organizing Map for Binary Data Topographic Clustering , 2008, Int. J. Comput. Intell. Appl..

[21]  Andreas Stafylopatis,et al.  A Hybrid Self-Organizing Model for Sequence Analysis , 2008, 2008 20th IEEE International Conference on Tools with Artificial Intelligence.

[22]  Andreas Stafylopatis,et al.  Sequence clustering with the Self-Organizing Hidden Markov Model Map , 2008, 2008 8th IEEE International Conference on BioInformatics and BioEngineering.

[23]  Djamel Bouchaffra,et al.  Embedding HMM’s-based models in a Euclidean space: The topological hidden Markov models , 2008, 2008 19th International Conference on Pattern Recognition.