Raw Speech Signal-based Continuous Speech Recognition using Convolutional Neural Networks

Abstract State-of-the-art automatic speech recognition systems model the relationship be-tween acoustic speech signal and phone classes in two stages, namely, extractionofspectral-basedfeaturesbasedonpriorknowledgefollowedbytrainingofacous-tic model, typically an articial neural network (ANN). In a recent work, it wasshownthatconvolutionneuralnetworks(CNNs)arecapableofmodelingtherela-tionbetweenacousticspeechsignalandphoneclassesdirectly. Thispaperextendsthe CNN-based approach to large vocabulary speech recognition task. More pre-cisely,wecomparetheCNN-basedapproachagainsttheconventionalANN-basedapproachonWallStreetJournalcorpus. OurstudiesshowthattheCNN-basedap-proachwithfewerparametersachievesperformancecomparableorbetterthantheconventional ANN-based approach. 1 Introduction State-of-the-art Automatic speech recognition (ASR) systems typically divide the task into severalsub-tasks, which are optimized in an independent manner [1]. In a rst step, the data is transformedinto features, usually composed of a dimensionality reduction phase and an information selectionphase, based on the task-specic knowledge of the phenomena. These two phases have been care-fully hand-crafted, leading to state-of-the-art features such as mel frequency cepstral coefcients(MFCCs) [2] or perceptual linear prediction cepstral features (PLPs) [3]. In a second step, the like-lihood of subword units such as, phonemes is estimated using generative models or discriminativemodels. In a nal step, dynamic programming techniques are used to recognize the word sequencegiven the lexical and syntactical constraints.Recent advances in machine learning have made possible systems that can be trained in an end-to-end manner, i.e. systems where every step islearned simultaneously, taking into account allthe other steps and the nal task of the whole system. It is typically referred to asdeep learning,mainly because such architectures are usually composed of many layers (supposed to provide anincreasing level of abstraction), compared to classical “shallow” systems. As opposed to “divideand conquer” approaches presented previously (where each step is independently optimized) deeplearning approaches are often claimed to have the potential to lead to more optimal systems, and tohave the advantage to alleviate the need of nd the right features for a given task of interest. Whilethere is a good success record of such approaches in the computer vision [4] or text processingelds [5], deep learning approaches for speech recognition still rely on spectral-based features suchas MFCC [6]. Some systems have proposed to learn features from “intermediate” representation ofspeech, like mel lter bank energies and their temporal derivatives.Inarecentstudy[7], itwasshownthatitispossibletoestimatephonemeclassconditionalprobabil-ities by using raw speech signal as input to convolutional neural networks [8] (CNNs). On TIMITphonemerecognitiontask, itwasshownthatthesystemisabletolearnfeaturesfromtherawspeech1