A Normalized Statistical Metric Space for Hidden Markov Models

In this paper, we present a normalized statistical metric space for hidden Markov models (HMMs). HMMs are widely used to model real-world systems. Like graph matching, some previous approaches compare HMMs by evaluating the correspondence, or goodness of match, between every pair of states, concentrating on the structure of the models instead of the statistics of the process being observed. To remedy this, we present a new metric space that compares the statistics of HMMs within a given level of statistical significance. Compared with the Kullback-Leibler divergence, which is another widely used approach for measuring model similarity, our approach is a true metric, can always return an appropriate distance value, and provides a confidence measure on the metric value. Experimental results are given for a sample application, which quantify the similarity of HMMs of network traffic in the Tor anonymization system. This application is interesting since it considers models extracted from a system that is intentionally trying to obfuscate its internal workings. In the conclusion, we discuss applications in less-challenging domains, such as data mining.

[1]  Satish T. S. Bukkapatnam,et al.  Zero knowledge hidden Markov model inference , 2009, Pattern Recognit. Lett..

[2]  Lalit R. Bahl,et al.  A Maximum Likelihood Approach to Continuous Speech Recognition , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  W. Press,et al.  Numerical Recipes in Fortran: The Art of Scientific Computing.@@@Numerical Recipes in C: The Art of Scientific Computing. , 1994 .

[4]  S. Wellek.  Testing Statistical Hypotheses of Equivalence , 2002 .

[5]  Gonzalo Navarro,et al.  A guided tour to approximate string matching , 2001, CSUR.

[6]  James Bennett,et al.  The Netflix Prize , 2007 .

[7]  Jeffrey D. Ullman,et al.  Formal languages and their relation to automata , 1969, Addison-Wesley series in computer science and information processing.

[8]  Byung-Jun Yoon,et al.  A Novel Low-Complexity HMM Similarity Measure , 2011, IEEE Signal Processing Letters.

[9]  Jason M. Schwier,et al.  Side-Channel Analysis for Detecting Protocol Tunneling , 2011, Adv. Internet Things.

[10]  C. Shalizi,et al.  Causal architecture, complexity and self-organization in time series and cellular automata , 2001 .

[11]  Bart De Moor,et al.  Equivalence of state representations for hidden Markov models , 2007, 2007 European Control Conference (ECC).

[12]  Jonathan L. Gross,et al.  Handbook of graph theory , 2007, Discrete mathematics and its applications.

[13]  K. Kiguchi,et al.  Modular fuzzy-neuro controller driven by spoken language commands , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[14]  Eric W. Weisstein,et al.  The CRC concise encyclopedia of mathematics , 1999 .

[15]  Jin Hyung Kim,et al.  Network-based approach to online cursive script recognition , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[16]  Jason M. Schwier,et al.  A Hybrid Statistical Technique for Modeling Recurrent Tracks in a Compact Set , 2011, IEEE Transactions on Automatic Control.

[17]  Bernadette Dorizzi,et al.  On Using the Viterbi Path Along With HMM Likelihood Information for Online Signature Verification , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[18]  Harikrishnan Bhanu,et al.  Timing Side-Channel Attacks on SSH , 2010 .

[19]  Jianping Zeng,et al.  A new distance measure for hidden Markov models , 2010, Expert Syst. Appl..

[20]  Ryan Craven,et al.  Traffic analysis of anonymity systems , 2010 .

[21]  Jason M. Schwier,et al.  Inferring Statistically Significant Hidden Markov Models , 2013, IEEE Transactions on Knowledge and Data Engineering.

[22]  Jason M. Schwier,et al.  Behavior Detection Using Confidence Intervals of Hidden Markov Models , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Zoubin Ghahramani,et al.  Learning Dynamic Bayesian Networks , 1997, Summer School on Neural Networks.

[24]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[25]  Joydeep Ghosh,et al.  Probabilistic model-based clustering of complex data , 2003 .

[26]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[27]  J. Neter,et al.  Applied Linear Regression Models , 1983 .

[28]  Richard R. Brooks,et al.  Pattern recognition for command and control data systems , 2009 .

[29]  Markus Falkhausen,et al.  Calculation of distance measures between hidden Markov models , 1995, EUROSPEECH.

[30]  Christopher D. Abbott.  Strengthening the Anonymity of Anonymous Communication Systems , 2011 .

[31]  Nick Mathewson,et al.  Tor: The Second-Generation Onion Router , 2004, USENIX Security Symposium.

[32]  Richard L. Scheaffer,et al.  Introduction to Probability and Its Applications. , 1991 .

[33]  Shrikanth S. Narayanan,et al.  Upper Bound Kullback–Leibler Divergence for Transient Hidden Markov Models , 2008, IEEE Transactions on Signal Processing.

[34]  L. R. Rabiner,et al.  A probabilistic distance measure for hidden Markov models , 1985, AT&T Technical Journal.

[35]  George S. Fishman,et al.  A First Course in Monte Carlo , 2005 .