Localized spectro-temporal features for automatic speech recognition

Recent results from physiological and psychoacoustic studies indicate that spectrally and temporally localized time-frequency envelope patterns form a relevant basis of auditory perception. This motivates new approaches to feature extraction for automatic speech recognition (ASR) which utilize two-dimensional spectro-temporal modulation filters. The paper provides a motivation and a brief overview on the work related to Localized Spectro-Temporal Features (LSTF). It further focuses on the Gabor feature approach, where a feature selection scheme is applied to automatically obtain a suitable set of Gabor-type features for a given task. The optimized feature sets are examined in ASR experiments with respect to robustness and their statistical properties are analyzed. 1. Getting auditory ... again? The question whether knowledge about the (human) auditory system provides valuable contributions to the design of ASR systems is as old as the field itself. The topic has been discussed extensively elsewhere (e.g. [1]). After all these years, a major argument still holds, namely the large gap in performance between normal-hearing native listeners and state-of-the art ASR systems. Consistently, humans outperform machines by at least an order of magnitude [2]. Human listeners recognize speech even in very adverse acoustical environments with strong reverberation and interfering sound sources. However, this discrepancy between human and machine performance is not restricted to robustness alone. It is observed also in undisturbed conditions and very small context independent corpora, where higher level constraints (cognitive aspects, language model) do not play a role. Arguably this hints towards an insufficient feature extraction in machine recognition systems. It is argued here, that including LSTF streams provides another step towards human-like speech recognition. 2. Evidence for (spectro-)temporal processing in the auditory system Speech is characterized by its fluctuations across time and frequency. The latter reflect the characteristics of the human vocal cords and tract and are commonly exploited in ASR by using short-term spectral representations such as cepstral coefficients. The temporal properties of speech are targeted in ASR by dynamic (delta and delta-delta) features and temporal filtering and feature extraction techniques like RASTA [3] and TRAPS [4]. Nevertheless, speech clearly exhibits combined spectro-temporal modulations. This is due to intonation, co-articulation and the succession of several phonetic elements, e.g., in a syllable. Formant transitions, for example, result in diagonal features in a spectrogram representation of speech. This kind of pattern is captured by LSTF and explicitly targeted by the Gabor feature extrac