Hierarchical Duration Modelling for a Speech Recognition System

Durational patterns of phonetic segments and pauses convey information about the linguistic content of an utterance. Most speech recognition systems grossly underutilize the knowledge provided by durational cues due to the vast array of factors that in uence speech timing and the complexity with which they interact. In this thesis, we introduce a duration model based on the Angie framework. Angie is a paradigm which captures morpho-phonemic and phonological phenomena under a uni ed hierarchical structure. Sublexical parse trees provided by Angie are well-suited for constructing complex statistical models to account for durational patterns that are functions of e ects at various linguistic levels. By constructing models for all the sublexical nodes of a parse tree, we implicitly model duration phenomena at these linguistic levels simultaneously, and subsequently account for a vast array of contextual variables a ecting duration from the phone level up to the word level. This thesis will describe our development of a durational model, and will demonstrate its utility in a series of experiments conducted in the Atis domain. The aim is to characterize phenomena such as speaking rate variability and prepausal lengthening in a quantitative manner. The duration model has been incorporated into a phonetic recognizer and a wordspotting system. We will report on the resulting improvement in performance. In this duration model, a strategy has been formulated in which node durations in upper layers are successively normalized by their respective realizations in the layers below; that is, given a nonterminal node, individual probability distributions, corresponding with each di erent realization in the layer immediately below, are all scaled to have the same mean. This reduces the variance at each node, and enables the sharing of statistical distributions. Upon normalization, a set of relative duration models is constructed by measuring the percentage duration of nodes occupied with respect to their parent nodes. Under this normalization scheme, the normalized duration of a word node is independent of the inherent durations of its descendents and hence is an indicator of speaking rate. A speaking rate parameter can be de ned as a ratio of the normalized word duration over the global average normalized word duration. This speaking rate parameter is then used to construct absolute duration models that are normalized by speaking rate. This is done by scaling either absolute phone or phoneme duration by the above parameter. By combining the hierarchical normalization and speaking rate normalization, the average standard deviation for phoneme duration was reduced from 50ms to 33ms. Using the hierarchical structure, we have conducted a series of experiments investigating speech timing phenomena. We are speci cally interested in the (1) examining secondary e ects of speaking rate, (2) characterizing the e ects of prepausal lengthening and (3) detecting other word boundary e ects associated with duration such as gemination. For example, we have found, with statistical signi cance, that a su x within a word is a ected far more by speaking rate than is a pre x. It is also observed that prepausal lengthening a ects the various sublexical units non-uniformly. For example, the stressed nucleus in the syllable tends to be lengthened more than the onset position. The nal duration model has been implemented into the Angie phonetic recognizer. In addition to contextual e ects captured by the model at various sublexical levels, the scoring mechanism also accounts explicitly for two inter-word level phenomena, namely, prepausal lengthening and gemination. Our experiments have been conducted under increasing levels of linguistic constraint with correspondingly di erent baseline performances. The improved performance is obtained by providing implicit lexical knowledge during recognition. When maximal linguistic contraint is imposed, the incorporation of the relative and speaking rate normalized absolute phoneme duration scores reduced the phonetic error rate from 29.7% to 27.4%, a relative reduction of 7.7%. These gains are over and above any gains realized from standard phone duration models present in the baseline system. As a rst step towards demonstrating the bene t of duration modelling for full word recognition, we have conducted a preliminary study using duration as a post-processor in a word-spotting task. We have simpli ed the task of spotting city names in the Atis domain by choosing a pair of highly confusable keywords, \New York" and \Newark." All tokens initially spotted as \New York" are passed to a post-processor, which reconsiders those words and makes a nal decision, with the duration component incorporated. For this task, the duration post-processor reduced the number of confusions from 60 to 19 tokens out of a total of 323 tokens, a 68% reduction of error. In another experiment, the duration model is fully integrated into an Angie-based wordspotting system. As in our phonetic recognition experiments, results were obtained at varying degrees of linguistic contraint. Here, when maximum constraint is imposed, the duration model improved performance from 89.3 to 91.6 (FOM), a relative improvement of 21.5%. This research has demonstrated success in employing a complex statistical duration model in order to improve speech recognition performance. It has shown that duration can play an important role in aiding word recognition and promises to o er greater gains for continuous word recognition. Thesis Supervisor: Stephanie Sene Title: Principal Research Scientist Hierarchical Duration Modelling for a Speech Recognition System