Margin-space integration of MPE loss via differencing of MMI functionals for generalized error-weighted discriminative training

Abstract Using the central observation that margin-based weightedclassification error (modeled using Minimum Phone Error(MPE)) corresponds to the derivative with respect to the mar-gin term of margin-based hinge loss (modeled using MaximumMutual Information (MMI)), this article subsumes and extendsmargin-based MPE and MMI within a broader framework inwhich the objective function is an integral of MPE loss over arange of margin values. Applying the Fundamental Theorem ofCalculus,thisintegraliseasilyevaluatedusingfinitedifferencesof MMI functionals; lattice-based training using the new crite-rion can then be carried out using differences of MMI gradi-ents. Experimental results comparing the new framework withmargin-based MMI, MCE and MPE on the Corpus of Sponta-neous Japanese and the MIT OpenCourseWare/MIT-World cor-pus are presented. 1. Introduction The field of discriminative training for speech recognition haswitnessed considerable activity in recent years. The appeal ofminimizingphoneorworderrorratherthanstringerrorhasmo-tivated a transition from well-known string-level methods suchas MMI and MCE [1][2] to error-weighted approaches, such asMPE [3][4]. More recently, there has been a surge in proposalsfor“largemargin”approachestohiddenMarkovmodel(HMM)design, such as the “large-margin HMM” [5], “soft margin es-timation” [6], and incrementally shifted MCE loss [7]. Sha andSaul [8] made the important proposal that a fine-grained er-ror measure, such as the Hamming distance between candidaterecognition strings, be itself directly incorporated into the mar-gin term for HMM-based learning. It turns out that introducinga margin term that multiplies fine-grained error can easily bebrought to MMI, MCE and MPE based HMM training as well,simply by adding margin-scaled local frame/phone/word errorto lattice arc log-likelihoods during Forward-Backward com-putation [9][10][11]. This approach links the original use ofmargin in the context of machine learning (e.g. Support VectorMachines (SVMs)) with margin in the context of “tried-and-tested” frameworks for large-scale discriminative training withwell-understood methods for HMM optimization on large-scaleASR tasks. Benefits to performance for large-scale tasks havebeen reported for the use of margin in MMI and MPE, thoughit appears the relative gains are larger for MMI than for MPE[10][11].Aiming at leveraging the benefits of margin use within thecontextofMPE-styleerror-weightedHMMtraining,thisarticlepresents a unification of margin-based MMI and MPE trainingbased on a novel concept: