Statistical Techniques for Language Recognition: an Introduction and Guide for Cryptanalysts

We explain how to apply statistical techniques to solve several language-recognition problems that arise in cryptanalysis and other domains. Language recognition is important in cryptanalysis because, among other applications, an exhaustive key search of any cryptosystem from ciphertext alone requires a test that recognizes valid plaintext. Written for cryptanalysts, this guide should also be helpful to others as an introduction to statistical inference on Markov chains. Modeling language as a finite stationary Markov process, we adapt a statistical model of pattern recognition to language recognition. Within this framework we consider four well-defined language-recognition problems: 1) recognizing a known language, 2) distinguishing a known language from uniform noise, 3) distinguishing unknown 0th-order noise from unknown lst-order language, and 4) detecting non-uniform unknown language. For the second problem we give a most powerful test based on the Neyman-Pearson Lemma. For the other problems, which ...

[1]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[2]  D. A. Bell,et al.  Information Theory and Reliable Communication , 1969 .

[3]  R. F.,et al.  Mathematical Statistics , 1944, Nature.

[4]  L. Stein,et al.  Probability and the Weighing of Evidence , 1950 .

[5]  J. F. Crook,et al.  The Powers and Strengths of Tests for Multinomials and Contingency Tables , 1982 .

[6]  William Frederick Friedman.  Military cryptanalytics. Part I , 1985 .

[7]  K. Liang,et al.  Asymptotic Properties of Maximum Likelihood Estimators and Likelihood Ratio Tests under Nonstandard Conditions , 1987 .

[8]  P. Billingsley,et al.  Statistical Methods in Markov Chains , 1961 .

[9]  Robert L. Solso,et al.  Bigram and trigram frequencies and versatilities in the English language , 1979 .

[10]  Carl A. Gunter,et al.  In handbook of theoretical computer science , 1990 .

[11]  B. John Oommen,et al.  Recognizing Sources of Random Strings , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  M. Bartlett.  The frequency goodness of fit test for probability chains , 1951, Mathematical Proceedings of the Cambridge Philosophical Society.

[13]  A. M. Turing,et al.  Studies in the History of Probability and Statistics. XXXVII A. M. Turing's statistical work in World War II , 1979 .

[14]  Solomon Kullback,et al.  Statistical Methods in Cryptanalysis , 1976 .

[15]  Connie Juel,et al.  THE FREQUENCY AND VERSATILITY OF INITIAL AND TERMINAL LETTERS IN ENGLISH WORDS , 1982 .

[16]  J. G. Gander,et al.  An introduction to signal detection and estimation , 1990 .

[17]  W. David Kelton,et al.  Hypothesis Tests for Markov Process Models Estimated from Aggregate Frequency Data , 1984 .

[18]  Kishor S. Trivedi.  Probability and Statistics with Reliability, Queuing, and Computer Science Applications , 1984 .

[19]  Irving John Good,et al.  The Estimation of Probabilities: An Essay on Modern Bayesian Methods , 1965 .

[20]  Robert L. Solso,et al.  Positional frequency and versatility of bigrams for two- through nine-letter English words , 1980 .

[21]  H. Seal.  Studies in the history of probability and statistics , 1977 .

[22]  I. Good.  Good Thinking: The Foundations of Probability and Its Applications , 1983 .

[23]  T. W. Anderson,et al.  Statistical Inference about Markov Chains , 1957 .

[24]  Yu-Chi Ho,et al.  On pattern classification algorithms--Introduction and survey , 1968 .

[25]  Sidney Yakowitz,et al.  Small-Sample Hypothesis Tests of Markov Order, with Application to Simulated and Hydrologic Chains , 1976 .

[26]  Harry L. Van Trees,et al.  Detection, Estimation, and Modulation Theory, Part I , 1968 .

[27]  I. J. Good,et al.  The fast calculation of the exact distribution of pearson's chi-squared and of the number of repeats within the cells of a mltltinomial by using a fast fourier transform , 1981 .

[28]  I. Good,et al.  Information, weight of evidence, the singularity between probability measures and signal detection , 1974 .

[29]  P. Hoel,et al.  A Solution to the Problem of Optimum Classification , 1949 .

[30]  Alan T. Sherman,et al.  How we Solved the $100, 000 Decipher Puzzle (16 Hours too late) , 1990, Cryptologia.

[31]  I. Good.  A Bayesian Significance Test for Multinomial Distributions , 1967 .

[32]  P. Holland,et al.  Discrete Multivariate Analysis. , 1976 .

[33]  DAVID G. KENDALL,et al.  Introduction to Mathematical Statistics , 1947, Nature.

[34]  W. Nelson Francis,et al.  FREQUENCY ANALYSIS OF ENGLISH USAGE: LEXICON AND GRAMMAR , 1983 .

[35]  A. Agresti.  [A Survey of Exact Inference for Contingency Tables]: Rejoinder , 1992 .

[36]  D. Kahn.  The codebreakers : the story of secret writing , 1968 .

[37]  J. F. Crook,et al.  The Bayes/Non-Bayes Compromise and the Multinomial Distribution , 1974 .

[38]  W. David Kelton,et al.  Development of Specific hypothesis tests for estimated markov chains , 1985 .

[39]  P. J. Weinberger,et al.  The UNIX system: File security and the UNIX system crypt command , 1984, AT&T Bell Laboratories Technical Journal.

[40]  D. Vere-Jones.  Markov Chains , 1972, Nature.

[41]  B. E. P. Clement,et al.  Automatic pattern recognition , 1992, Defense, Security, and Sensing.

[42]  H. Kucera,et al.  Computational analysis of present-day American English , 1967 .

[43]  J. Dickey,et al.  The Weighted Likelihood Ratio, Sharp Hypotheses about Chances, the Order of a Markov Chain , 1970 .

[44]  Kumpati S. Narendra,et al.  Adaptive, learning, and pattern recognition systems: Theory and applications , 1972 .

[45]  Naftali Z. Tisby.  On the application of mixture AR hidden Markov models to text independent speaker recognition , 1991, IEEE Trans. Signal Process..

[46]  R. Larsen.  An introduction to mathematical statistics and its applications / Richard J. Larsen, Morris L. Marx , 1986 .

[47]  H. Vincent Poor,et al.  An Introduction to Signal Detection and Estimation , 1994, Springer Texts in Electrical Engineering.

[48]  Hsiao-Wuen Hon,et al.  Speaker-independent phone recognition using hidden Markov models , 1989, IEEE Trans. Acoust. Speech Signal Process..

[49]  Claude E. Shannon,et al.  Communication theory of secrecy systems , 1949, Bell Syst. Tech. J..

[50]  James M. Dickey,et al.  Discussion: Testing for Independence in a Two-Way Table: New Interpretations of the Chi-Square Statistic , 1985 .

[51]  Gustavus J. Simmons,et al.  Contemporary Cryptology: The Science of Information Integrity , 1994 .

[52]  S. Kullback,et al.  Tests for Contingency Tables and Marltov Chains , 1962 .

[53]  Henry Beker,et al.  Cipher Systems: The Protection of Communications , 1982 .

[54]  I. Good.  The Bayes/Non-Bayes Compromise: A Brief Review , 1992 .

[55]  Patrick Billingsley,et al.  Statistical inference for Markov processes , 1961 .

[56]  Andri Ariste,et al.  Pattern analysis and understanding , 1990 .

[57]  Josef Raviv,et al.  Decision making in Markov chains applied to the problem of pattern recognition , 1967, IEEE Trans. Inf. Theory.

[58]  Alan T. Sherman,et al.  Statistical Techniques for Language Recognition: an Empirical Study Using Real and Simulated English , 1994, Cryptologia.

[59]  Ronald L. Rivest.  Statistical Analysis of the Hagelin Cryptograph , 1981, Cryptologia.

[60]  I. Good.  On the Application of Symmetric Dirichlet Distributions and their Mixtures to Contingency Tables , 1976 .

[61]  I. J. Good,et al.  The Frequency Count of a Markov Chain and the Transition to Continuous Time , 1961 .

[62]  Stephen E. Levinson,et al.  Development of an acoustic-phonetic hidden Markov model for continuous speech recognition , 1991, IEEE Trans. Signal Process..

[63]  William F. Friedman.  The index of coincidence and its applications in cryptanalysis , 1987 .

[64]  Rafael Hirschfeld,et al.  Pseudorandom Generators and Complexity Classes , 1989, Advances in Computational Research.

[65]  Vikram Krishnamurthy,et al.  Hidden Markov Model Signal Processing in Presence , 1996 .

[66]  Aaron D. Wyner,et al.  Prediction and Entropy of Printed English , 1993 .

[67]  Manuel Blum,et al.  Towards a Computational Theory of Statistical Tests (Extended Abstract) , 1992, FOCS 1992.

[68]  HE Ixtroductiont,et al.  The Bell System Technical Journal , 2022 .

[69]  Frank A. Feldman.  A New Spectral Test for Nonrandomness and the DES , 1990, IEEE Trans. Software Eng..

[70]  James A. Reeds,et al.  On the cryptanalysis of rotor machines and substitution - permutation networks , 1982, IEEE Trans. Inf. Theory.

[71]  Dorothy E. Denning,et al.  Cryptography and Data Security , 1982 .

[72]  Robert L. Solso,et al.  Frequency and versatility of letters in the English language , 1976 .

[73]  Irving John Good,et al.  Exact Distributions for χ2 and for the Likelihood-Ratio Statistic for the Equiprobable Multinomial Distribution , 1970 .

[74]  L. R. Rabiner,et al.  A probabilistic distance measure for hidden Markov models , 1985, AT&T Technical Journal.

[75]  CipherThomas R. Cain,et al.  How to Break Giiord's Cipher , 1994 .