Robust Automatic Speech Recognition with Missing and Unreliable Data

Automatic speech recognition (ASR) systems have made dramatic performance leaps in the recent past. Yet, the notion that the key to making recognition more robust is to reduce the difference between training and test conditions is still commonly held. As ASR applications move from tightly controlled to more natural environments with a varying number of unpredictable sound sources, this assumption is becoming less and less viable. Decoding the speech source of interest while listening to several sound sources at the same time seems a more accurate description of the ASR process that suits these challenging environments. This thesis discusses the theoretical and practical issues which arise from this viewpoint. The aim is to explore the division of the problem of robust ASR into two subproblems: (a) identification/separation of the speech and noise using speech properties alone; and (b) recognition based on the resulting partial evidence. The basic assumption is that some regions of the speech time-frequency representation remain relatively unaffected by the noise, that they can be identified and that they alone are sufficient for ASR. In contrast to conventional techniques which require models of all sources in the auditory scene and their subsequent decoding even when only one of the sources is of interest, the techniques described in this thesis make no such requirement. However, they are flexible enough to use this information if it is available. Two techniques are used to adapt a conventional Hidden Markov model (HMM) based ASR system to use partial evidence: (i) marginalisation of the state distributions, so that only the likelihood of the reliable regions is assessed; and (ii) imputation of the unreliable regions by replacing the unreliable features with a single point from the state conditional distributions. In both cases, the ”counterevidence” assessing which states are unlikely to have generated the speech underlying the unreliable regions dominated by noise further constrains the decoding. The techniques are evaluated on the Aurora 2 connected digit recognition task, and seem to perform competitively. In the experiments, the reliable features are identified via local SNR estimates derived through stationary and adaptive on-line noise estimates. The potential of the techniques is indicated by using the clean speech to identify the reliable regions in the noisy speech, where the accuracy is maintained even at -5 dB. The simple all-or-nothing assumption (the feature is either reliable or unreliable) gives rise to a model linking the recognition and the separation as two interdependent sides of the search for the most likely explanation of the noisy data.