Online Speaker Adaptation of an Acoustic Model Using Face Recognition

We have proposed and evaluated a novel approach for online speaker adaptation of an acoustic model based on face recognition. Instead of traditionally used audio-based speaker identification we investigated the video modality for the task of speaker detection. A simulated on-line transcription created by a Large-Vocabulary Continuous Speech Recognition (LVCSR) system for online subtitling is evaluated utilizing speaker independent acoustic models, gender dependent models and models of particular speakers. In the experiment, the speaker dependent acoustic models were trained offline, and are switched online based on the decision of a face recognizer, which reduced Word Error Rate (WER) by 12% relatively compared to speaker independent baseline system.

[1]  Sören Sonnenburg,et al.  Optimized Cutting Plane Algorithm for Large-Scale Risk Minimization , 2009, J. Mach. Learn. Res..

[2]  Philippe Joly,et al.  Audiovisual diarization of people in video content , 2012, Multimedia Tools and Applications.

[3]  Sören Sonnenburg,et al.  COFFIN: A Computational Framework for Linear SVMs , 2010, ICML.

[4]  Hervé Bourlard,et al.  Robust speaker change detection , 2004, IEEE Signal Processing Letters.

[5]  Zbynek Zajíc,et al.  Fast Speaker Adaptation in Automatic Online Subtitling , 2009, SIGMAP.

[6]  Václav Hlavác,et al.  Detector of Facial Landmarks Learned by the Structured Output SVM , 2012, VISAPP.

[7]  Roman Grundkiewicz,et al.  Automatic Extraction of Polish Language Errors from Text Edition History , 2013, TSD.

[8]  Josef Psutka,et al.  Online TV Captioning of Czech Parliamentary Sessions , 2010, TSD.

[9]  Jonathan Le Roux,et al.  Discriminative Training for Large-Vocabulary Speech Recognition Using Minimum Classification Error , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[10]  Ludek Müller,et al.  Automatic Online Subtitling of the Czech Parliament Meetings , 2006, TSD.

[11]  Ludek Müller,et al.  Robust Statistic Estimates for Adaptation in the Task of Speech Recognition , 2010, TSD.

[12]  Jan Vanek,et al.  Speaker-Clustered Acoustic Models Evaluated on GPU for On-line Subtitling of Parliament Meetings , 2011, TSD.

[13]  Gérard Chollet,et al.  People indexing in TV-content using lip-activity and unsupervised audio-visual identity verification , 2011, 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI).