Enhancing speech recognition in fast-paced educational games using contextual cues

Arcade-style games like Tetris and Pacman are often difficult to adapt for educational purposes because their fast-paced intensity and keystroke-heavy nature leave little room for simultaneous practice of other skills. Incorporating spoken language technology could make it possible for players to learn as they play, keeping up with game speed through multimodal interaction. To date, however, it remains exceedingly difficult to augment fast-paced games with speech interaction because the frustrating effect of recognition errors highly compromises entertainment. In this paper, we design a modified version of Tetris with speech recognition to help students practice and remember word-picture mappings. Using utterances collected from learners interacting with the speechenabled Tetris game, we present and evaluate several techniques for leveraging contextual cues to increase recognition accuracy in fast-paced game environments.

[1]  Joseph Polifroni,et al.  Recognition confidence scoring and its use in speech understanding systems , 2002, Comput. Speech Lang..

[2]  C. Bisson,et al.  Fun in Learning: The Pedagogical Role of Fun in Adventure Education , 1996 .

[3]  Wei Xu,et al.  Language modeling for dialog system , 2000, INTERSPEECH.

[4]  James R. Glass,et al.  Learning Lexicons From Speech Using a Pronunciation Mixture Model , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[5]  M. Csíkszentmihályi,et al.  Flow Theory and Research , 2009 .

[6]  Hermann Ney,et al.  Joint-sequence models for grapheme-to-phoneme conversion , 2008, Speech Commun..

[7]  James R. Glass,et al.  Exploiting Context Information in Spoken Dialogue Interaction with Mobile Devices ? , 2022 .

[8]  James R. Glass.  A probabilistic framework for segment-based speech recognition , 2003, Comput. Speech Lang..

[9]  Matthew Kam,et al.  Improving literacy in developing countries using speech recognition-supported games on mobile devices , 2012, CHI.

[10]  Roger K. Moore.  Computer Speech and Language , 1986 .

[11]  Stephanie Seneff,et al.  Speech-enabled card games for incidental vocabulary acquisition in a foreign language , 2009, Speech Commun..

[12]  Herre van Oostendorp,et al.  Code Red: Triage, Or, COgnition-based DEsign Rules Enhancing Decisionmaking TRaining In A Game Environment , 2011 .

[13]  Stephanie Seneff,et al.  Improving nonnative speech understanding using context and N-best meaning fusion , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  S. Wilson.  What Video Games Have to Teach Us about Learning and Literacy , 2006 .

[15]  Thomas W. Malone,et al.  What Makes Things Fun to Learn? A Study of Intrinsically Motivating Computer Games. , 1981 .

[16]  Ian McGraw,et al.  The WAMI toolkit for developing, deploying, and evaluating web-accessible multimodal interfaces , 2008, ICMI '08.

[17]  Carrie J. Cai.  Adapting arcade games for learning , 2013, CHI Extended Abstracts.