Conversational agents in virtual worlds: Bridging disciplines

This paper examines the effective deployment of conversational agents in virtual worlds from the perspective of researchers/practitioners in cognitive psychology, computing science, learning technologies and engineering. From a cognitive perspective, the major challenge lies in the coordination and management of the various channels of information associated with conversation/communication and integrating this information with the virtual space of the environment and the belief space of the user. From computing science, the requirements include conversational competency, use of nonverbal cues, animation consistent with affective states, believability, domain competency and user adaptability. From a learning technologies perspective, the challenge is to maximise the considerable affordances provided by conversational avatars in virtual worlds balanced against ecologically valid investigations regarding utility. Finally, the engineering perspective focuses on the technical competency required to implement effective and functional agents, and the associated costs to enable student access. Taken together, the four perspectives draw attention to the quality of the agent–user interaction, how theory, practice and research are closely intertwined, and the multidisciplinary nature of this area with opportunities for cross fertilisation and collaboration. [ABSTRACT FROM AUTHOR]

[1]  R. Mayer,et al.  Learning Science in Virtual Reality Multimedia Environments: Role of Methods and Media , 2002 .

[2]  Mel Slater,et al.  Spatial Social Behavior in Second Life , 2007, IVA.

[3]  Jason Nolan,et al.  Hacking Say and Reviving ELIZA: Lessons from Virtual Environments , 2009 .

[4]  Jan-Willem Strijbos,et al.  Designing electronic collaborative learning environments , 2004 .

[5]  Gary R. Bradski,et al.  Motion segmentation and pose recognition with motion history gradients , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[6]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[7]  Robin Kay A Systematic Evaluation of Learning Objects for Secondary School Students , 2007 .

[8]  Xiang Yuan,et al.  Design and evaluation of Elva: an embodied tour guide in an interactive virtual art gallery , 2005, Comput. Animat. Virtual Worlds.

[9]  Magnus Haake,et al.  Visual Stereotypes and Virtual Pedagogical Agents , 2008, J. Educ. Technol. Soc..

[10]  Tamás Szirányi,et al.  Hand Gesture Recognition in Camera-Projector System , 2004, ECCV Workshop on HCI.

[11]  Magnus Haake,et al.  Embodied Pedagogical Agents: From Visual Impact to Pedagogical Implications , 2009 .

[12]  Punya Mishra,et al.  RUNNING HEAD: PEDAGOGICAL SOFTWARE AGENTS All or Nothing: Levels of Sociability of a Pedagogical Software Agent and its Impact on Student Perceptions and Learning , 2005 .

[13]  Yanghee Kim,et al.  Pedagogical Agents as Learning Companions: The Role of Agent Competency and Type of Interaction , 2006 .

[14]  S. Greenberg,et al.  The Psychology of Everyday Things , 2012 .

[15]  Patricia D. Mautone,et al.  Social cues in multimedia learning: Role of speaker's voice. , 2003 .

[16]  Aaron Doering,et al.  When sex, drugs, and violence enter the classroom: Conversations between adolescents and a female pedagogical agent , 2008, Interact. Comput..

[17]  Kristinn R. Thórisson,et al.  The Power of a Nod and a Glance: Envelope Vs. Emotional Feedback in Animated Conversational Agents , 1999, Appl. Artif. Intell..

[18]  Marion Langer,et al.  Virtual humans with personalized perception and dynamic levels of knowledge , 2005, Comput. Animat. Virtual Worlds.

[19]  M. Betke,et al.  The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Arthur C. Graesser,et al.  AUTOMATIC DETECTION OF LEARNER'S AFFECT FROM GROSS BODY LANGUAGE , 2009, Appl. Artif. Intell..

[21]  Susanne van Mulken,et al.  The impact of animated interface agents: a review of empirical research , 2000, Int. J. Hum. Comput. Stud..

[22]  Efthymios Alepis,et al.  Requirements Analysis and Design of an Affective Bi-Modal Intelligent Tutoring System: The Case of Keyboard and Microphone , 2008 .

[23]  Christopher Hoadley,et al.  Learning sciences and instructional systems: Beginning the dialogue (Special issue] , 2004 .

[24]  Kevin O'Brien,et al.  Human Behavior Models for Agents in Simulators and Games: Part I: Enabling Science with PMFserv , 2006, Presence: Teleoperators & Virtual Environments.

[25]  Donald A. Norman,et al.  How might people interact with agents , 1994, CACM.

[26]  George Veletsianos,et al.  The impact and implications of virtual character expressiveness on learning and agent-learner interactions , 2009, J. Comput. Assist. Learn..

[27]  Amy L. Baylor,et al.  Permutations of Control: Cognitive Considerations for Agent-Based Learning Environments , 2001 .

[28]  Agneta Gulz,et al.  Benefits of Virtual Characters in Computer Based Learning Environments: Claims and Evidence , 2004, Int. J. Artif. Intell. Educ..

[29]  Susan Bull,et al.  Bringing chatbots into education: Towards natural language negotiation of open learner models , 2006, Knowl. Based Syst..

[30]  Design-Based Research: An Emerging Paradigm for Educational Inquiry , 2003 .

[31]  Michael J. Singer,et al.  Virtual environment cultural training for operational readiness (VECTOR) , 2004, Virtual Reality.

[32]  Sheila B. Banks,et al.  Progress and Prospects for the Development of Computer-Generated Actors for Military Simulation: Part 1-Introduction and Background , 2003, Presence: Teleoperators & Virtual Environments.

[33]  Robert Heller,et al.  Animated Pedagogical Agents: The Effect of Visual Information on a Historical Figure Application , 2009, Int. J. Web Based Learn. Teach. Technol..

[34]  Magnus Haake,et al.  Design of animated pedagogical agents - A look at their look , 2006, Int. J. Hum. Comput. Stud..

[35]  Yanghee Kim,et al.  Desirable Characteristics of Learning Companions , 2007, Int. J. Artif. Intell. Educ..

[36]  Mubarak Shah,et al.  A virtual 3D blackboard: 3D finger tracking using a single camera , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[37]  Steven Warburton,et al.  Second Life in higher education: Assessing the potential for and the barriers to deploying virtual worlds in learning and teaching , 2009, Br. J. Educ. Technol..

[38]  Paul Nielsen,et al.  Participation of TacAir-Soar in RoadRunner and Coyote Exercises at Air Force Research Lab, Mesa AZ , 2006 .

[39]  George Veletsianos Cognitive and Affective Benefits of an Animated Pedagogical Agent: Considering Contextual Relevance and Aesthetics , 2007 .

[40]  Ralph E. Chatham,et al.  Games for training , 2007, CACM.

[41]  Charles Miller,et al.  Conversing with pedagogical agents: A phenomenological exploration of interacting with digital entities , 2008, Br. J. Educ. Technol..

[42]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[43]  Hans Uszkoreit,et al.  Conversational Agents in a Virtual World , 2011, KI.

[44]  Scotty D. Craig,et al.  Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy , 2002 .

[45]  Cristina Conati,et al.  Probabilistic assessment of user's emotions in educational games , 2002, Appl. Artif. Intell..

[46]  Abdolhossein Sarrafzadeh,et al.  "How do you know that I don't understand?" A look at the future of intelligent tutoring systems , 2008, Comput. Hum. Behav..

[47]  Rosalind W. Picard,et al.  An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[48]  Sheila B. Banks,et al.  Progress and Prospects for the Development of Computer Generated Actors for Military Simulation, Part 3The Road Ahead , 2003, Presence: Teleoperators & Virtual Environments.

[49]  Zhen Liu,et al.  Cognitive and synthetic behavior of avatars in intelligent virtual environments , 2008, Virtual Reality.

[50]  Arthur C. Graesser,et al.  Automatic detection of learner’s affect from conversational cues , 2008, User Modeling and User-Adapted Interaction.

[51]  Bonnie M Dobbs,et al.  Video Programming for Individuals With Dementia: Assessing Cognitive Congruence , 2009, American journal of Alzheimer's disease and other dementias.

[52]  A. Collins,et al.  Situated Cognition and the Culture of Learning , 1989 .

[53]  Yam San Chee,et al.  Design and evaluation of Elva: an embodied tour guide in an interactive virtual art gallery: Research Articles , 2005 .

[54]  Peter Abdelmassih Waller,et al.  Extended Television: A Study of How Investigations of Use Can Inform Design Processes in Nursing Homes , 2007, PsychNology J..

[55]  R. Sternberg What Is an “Expert Student?” , 2003 .

[56]  Aaron Doering,et al.  Enali: A Research and Design Framework for Virtual Characters and Pedagogical Agents , 2009 .

[57]  Rosalind W. Picard,et al.  Affective learning companions: strategies for empathetic agents with real-time multimodal affective sensing to foster meta-cognitive and meta-affective approaches to learning, motivation, and perseverance , 2006 .