Modelling Spoken Multimodal Instructional Systems

The use of speech and spoken dialogue is a relatively recent addition to instructional systems. As, almost invariably, human instructors and students talk during teaching and training, spoken dialogue would seem to be an important factor in systems that emulate aspects of human instruction. In this chapter, the origins and state of the art of spoken multimodal instruction are descrbed. Strengths and weaknesses of the speech modality, key roles of spoken dialogue in multimodal instruction, functional issues in current spoken teaching and training systems, commercial prospects, and some main challenges ahead are then discussed. IntroductIon A key advantage of instructional systems is to enable instruction in the absence of a human expert or teacher. From pre-school kids to adults of all ages, everybody needs to learn and benefit from the expertise of others when doing unfamiliar tasks. The classical solution is to be helped by a human instructor who has two kinds of expertise: in the subject-matter in question and in effectively communicating or transferring the expertise to students. While this approach has worked for millennia, it suffers from the problem that expertise remains expensive and rare, relative to the number of those who wish to acquire or draw upon it. A language instructor in class, for instance, has little time for coaching each student individually. Niels Ole Bernsen NISLab, University of Southern Denmark, Denmark Laila Dybkjær NISLab, University of Southern Denmark, Denmark Modelling Spoken Multimodal Instructional Systems

[1]  Terry Winograd,et al.  Procedures As A Representation For Data In A Computer Program For Understanding Natural Language , 1971 .

[2]  Susanne P. Lajoie,et al.  Sherlock: A Coached Practice Environment for an Electronics Troubleshooting Job. , 1988 .

[3]  Masahiro Araki,et al.  Spoken, Multilingual and Multimodal Dialogue Systems: Development and Assessment , 2005 .

[4]  James F. Allen A plan-based approach to speech act recognition , 1979 .

[5]  Niels Ole Bernsen,et al.  Multimodality in Language and Speech Systems — From Theory to Design Support Tool , 2002 .

[6]  Bruce Roberts Coaching Driving Skills in a Shiphandling Trainer , 2002 .

[7]  Ronnie W. Smith,et al.  A computational model of expectation-driven mixed-initiative dialog processing , 1992 .

[8]  Michael F. McTear,et al.  Spoken Dialogue Technology , 2004, Springer London.

[9]  Alexander I. Rudnicky,et al.  Sorry, I Didn’t Catch That! , 2008 .

[10]  Julia Hirschberg,et al.  Detecting certainness in spoken tutorial dialogues , 2005, INTERSPEECH.

[11]  Randall W. Hill,et al.  Virtual Humans in the Mission Rehearsal Exercise System , 2003, Künstliche Intell..

[12]  Niels Ole Bernsen,et al.  Cooperativity in human‐machine and human‐human spoken dialogue , 1996 .

[13]  Catherine Pelachaud,et al.  From brows to trust: evaluating embodied conversational agents , 2004 .

[14]  Gautam Biswas,et al.  LEARNING BY TEACHING: A NEW AGENT PARADIGM FOR EDUCATIONAL SOFTWARE , 2005, Appl. Artif. Intell..

[15]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[16]  S. Pressey A machine for automatic teaching of drill material. , 1927 .

[17]  Heather H. Mitchell,et al.  AutoTutor: A tutor with dialogue in natural language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[18]  Carolyn Penstein Rosé,et al.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing , 2002, Intelligent Tutoring Systems.

[19]  Sandra Katz,et al.  Extensions of intelligent tutoring paradigms to support collaborative learning , 1992 .

[20]  Barbara G Deutsch,et al.  The Structure of Task Oriented Dialogs , 1974 .

[21]  Diane J. Litman,et al.  Dependencies between Student State and Speech Recognition Problems in Spoken Tutoring Dialogues , 2006, ACL.

[22]  David R. Traum,et al.  Book Reviews: Spoken Natural Language Dialogue Systems: A Practical Approach , 1996, CL.

[23]  Dominic W. Massaro,et al.  The Psychology and Technology of Talking Heads: Applications in Language Learning , 2005 .

[24]  Niels Ole Bernsen,et al.  Combining multi-party speech and text exchanges over the internet , 2001, INTERSPEECH.

[25]  Stacy Marsella,et al.  Virtual Humans for non-team interaction training , 2005, AAMAS 2005.

[26]  Arthur C. Graesser,et al.  Intelligent Tutoring Systems with Conversational Dialogue , 2001, AI Mag..

[27]  Niels Ole Bernsen,et al.  Designing Interactive Speech Systems , 1998, Springer London.

[28]  Stanley Peters,et al.  Intelligent Systems for Training Damage Control Assistants , 2004 .

[29]  Niels Ole Bernsen,et al.  Meet Hans Christian Andersen , 2005, SIGDIAL Workshop.

[30]  Niels Ole Bernsen,et al.  Field Evaluation of a Single-Word Pronunciation Training System , 2006, LREC.

[31]  Diane J. Litman,et al.  Speech recognition performance and learning in spoken dialogue tutoring , 2005, INTERSPEECH.

[32]  Allan Collins,et al.  The goal structure of a socratic tutor , 1977, ACM Annual Conference.

[33]  J. V. Kuppevelt,et al.  Advances in natural multimodal dialogue systems , 2005 .

[34]  S. Dijkstra,et al.  Instructional Models in Computer-Based Learning Environments , 1992 .

[35]  Joel R. Tetreault,et al.  Using system and user performance features to improve emotion detection in spoken tutoring dialogs , 2006, INTERSPEECH.

[36]  Claus Zinn,et al.  The Role of Initiative in Tutorial Dialogue , 2003, EACL.

[37]  H. Grice Logic and conversation , 1975 .

[38]  Niels Ole Bernsen,et al.  Fusion of children's speech and 2D gestures when conversing with 3D characters , 2006, Signal Process..

[39]  Diane J. Litman,et al.  ITSPOKE: An Intelligent Tutoring Spoken Dialogue System , 2004, NAACL.

[40]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[41]  Candace L. Sidner,et al.  COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction , 2001, AI Mag..

[42]  Michael F. McTear,et al.  Book Review , 2005, Computational Linguistics.

[43]  Elisabeth André,et al.  Perception in Multimodal Dialogue Systems, 4th IEEE Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems, PIT 2008, Kloster Irsee, Germany, June 16-18, 2008, Proceedings , 2008, PIT.

[44]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[45]  Joel R. Tetreault,et al.  Comparing Synthesized versus Pre-Recorded Tutor Speech in an Intelligent Tutoring Spoken Dialogue System , 2006, FLAIRS.

[46]  Niels Ole Bernsen,et al.  Advances in Natural Multimodal Dialogue Systems (Text, Speech and Language Technology) , 2007 .

[47]  Candace L. Sidner,et al.  Collaborative Discourse Theory as a Foundation for Tutorial Dialogue , 2002, Intelligent Tutoring Systems.

[48]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[49]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[50]  Laila Dybkjær,et al.  Recent trends in discourse and dialogue , 2008 .

[51]  D. Murray Learning by Teaching , 1982 .

[52]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[53]  Ronald A. Cole,et al.  Perceptive animated interfaces: first steps toward a new paradigm for human-computer interaction , 2003, Proc. IEEE.

[54]  Niels Ole Bernsen,et al.  Conversational H.C. Andersen First Prototype Description , 2004, ADS.

[55]  A. Graesser,et al.  LEARNING WHILE HOLDING A CONVERSATION WITH A COMPUTER , 2005 .

[56]  Alexander I. Rudnicky,et al.  Sorry and I Didn’t Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies , 2005, SIGDIAL.

[57]  Carolyn Penstein Rosé,et al.  Spoken Versus Typed Human and Computer Dialogue Tutoring , 2006, Int. J. Artif. Intell. Educ..

[58]  Niels Ole Bernsen,et al.  Designing interactive speech systems - from first ideas to user testing , 1998 .

[59]  Niels Ole Bernsen,et al.  Multimodal Usability , 2010, Human-Computer Interaction Series.

[60]  Niels Ole Bernsen,et al.  Towards a tool for predicting speech functionality , 1997, Speech Commun..

[61]  Diane J. Litman,et al.  Modelling User Satisfaction and Student Learning in a Spoken Dialogue Tutoring System with Generic, Tutoring, and User Affect Parameters , 2006, NAACL.