Contextual Analysis for Middle Eastern Languages with Hidden Markov Models

Displaying a document in Middle Eastern languages requires contextual analysis due to different presentational forms for each character of the alphabet. The words of the document will be formed by the joining of the correct positional glyphs representing corresponding presentational forms of the characters. A set of rules defines the joining of the glyphs. As usual, these rules vary from language to language and are subject to interpretation by the software developers. In this paper, we propose a machine learning approach for contextual analysis based on the first order Hidden Markov Model. We will design and build a model for the Farsi language to exhibit this technology. The Farsi model achieves 94% accuracy with the training based on a short list of 89 Farsi vocabularies consisting of 2780 Farsi characters. The experiment can be easily extended to many languages including Arabic, Urdu, and Sindhi. Furthermore, the advantage of this approach is that the same software can be used to perform contextual analysis without coding complex rules for each specific language. Of particular interest is that the languages with fewer speakers can have greater representation on the web, since they are typically ignored by software developers due to lack of financial incentives.

[1]  Phil Blunsom,et al.  Compositional Morphology for Word Representations and Language Modelling , 2014, ICML.

[2]  Kazem Taghva,et al.  A stemming algorithm for the Farsi language , 2005, International Conference on Information Technology: Coding and Computing (ITCC'05) - Volume II.

[3]  Djoerd Hiemstra,et al.  Challenges in information retrieval and language modeling: report of a workshop held at the center for intelligent information retrieval, University of Massachusetts Amherst, September 2002 , 2003, SIGF.

[4]  Kazem Taghva,et al.  Language model-based retrieval for Farsi documents , 2004, International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004..

[5]  Kazem Taghva,et al.  Arabic stemming without a root dictionary , 2005, International Conference on Information Technology: Coding and Computing (ITCC'05) - Volume II.

[6]  Herbert Penzl.  A Grammar of Pashto A Descriptive Study of the Dialect of Kandahar, Afghanistan , 2009 .

[7]  Alex Waibel,et al.  Readings in speech recognition , 1990 .

[8]  Xing M. Wang.  Probability Bracket Notation: Markov State Chain Projector, Hidden Markov Models and Dynamic Bayesian Networks , 2012, ArXiv.

[9]  Tim Bray.  Element Sets: A Minimal Basis for an XML Query Engine , 1998, QL.

[10]  Kazem Taghva,et al.  Farsi searching and display technologies , 2003 .

[11]  Yang He.  Extended Viterbi algorithm for second order hidden Markov process , 1988, [1988 Proceedings] 9th International Conference on Pattern Recognition.

[12]  Robert L. Mercer,et al.  The Mathematics of Statistical Machine Translation: Parameter Estimation , 1993, CL.

[13]  Kazem Taghva,et al.  Post processing with first- and second-order hidden Markov models , 2013, Electronic Imaging.

[14]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.