Social fidelity in virtual agents: Impacts on presence and learning

Abstract Teaching and training are increasingly moving from real world venues to computerized environments, with human instructors often being replaced or joined by virtual pedagogical agents. While system fidelity and immersive properties of virtual learning environments are frequently discussed in the literature, less often addressed is the fidelity of the social components of teaching and their inclusion in pedagogical agent design. Teaching is inherently a social process, making the social fidelity of virtual agents a potential factor affecting learning outcomes. In this paper, we explore the concept of social fidelity as it pertains to the teaching effectiveness of pedagogical agents. We define the term, distinguish two subcategories, and discuss representative examples of research in these domains. Promising avenues for improving learning outcomes with social fidelity include personalized language, politeness, personality, attention, feedback, social memory, and gestures. Key conclusions are that: Social fidelity is important to learning in non-social domains, tradeoffs exist when implementing certain forms of social fidelity, individual user differences need to be more widely considered, and more focused studies are needed to compare different levels of social fidelity to uncover how they impact learning outcomes.

[1]  A. L. Baylor,et al.  The Effects of Pedagogical Agent Voice and Animation on Learning, Motivation and Perceived Persona , 2003 .

[2]  Anthony G. Picciano BEYOND STUDENT PERCEPTIONS: ISSUES OF INTERACTION, PRESENCE, AND PERFORMANCE IN AN ONLINE COURSE , 2019, Online Learning.

[3]  Shawn A. Weil,et al.  From Gaming to Training: A Review of Studies on Fidelity, Immersion, Presence, and Buy-in and Their Effects on Transfer in PC-Based Simulations and Games , 2005 .

[4]  Scotty D. Craig,et al.  Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy , 2002 .

[5]  R. Mayer The Cambridge Handbook of Multimedia Learning: Principles Based on Social Cues in Multimedia Learning: Personalization, Voice, Image, and Embodiment Principles , 2014 .

[6]  James C. Lester,et al.  Life-Like Pedagogical Agents in Constructivist Multimedia Environments: Cognitive Consequences of their Interaction , 2000 .

[7]  D. Garrison ONLINE COMMUNITY OF INQUIRY REVIEW: SOCIAL, COGNITIVE, AND TEACHING PRESENCE ISSUES , 2019, Online Learning.

[8]  Jennifer C. Richardson,et al.  EXAMINING SOCIAL PRESENCE IN ONLINE COURSES IN RELATION TO STUDENTS' PERCEIVED LEARNING AND SATISFACTION , 2003, Online Learning.

[9]  John J. Foxe,et al.  Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. , 2006, Cerebral cortex.

[10]  Benjamin Kehrwald,et al.  Understanding social presence in text‐based online learning environments , 2008 .

[11]  Shannon K. T. Bailey,et al.  The Impact of Synthetic and Accented Speech on Unattended Recall in a Dichotic Listening Task , 2012 .

[12]  Parham Mokhtari,et al.  On the human ability to auditorily perceive human speaker's facing angle , 2010, 2010 4th International Universal Communication Symposium.

[13]  Richard E. Mayer,et al.  Animated Pedagogical Agents as Aids in Multimedia Learning: Effects on Eye-Fixations During Learning and Learning Outcomes , 2017 .

[14]  Heiga Zen,et al.  The HTS-2008 System: Yet Another Evaluation of the Speaker-Adaptive HMM-based Speech Synthesis System in The 2008 Blizzard Challenge , 2008 .

[15]  Steve Nebel,et al.  Introducing the familiarity mechanism: A unified explanatory approach for the personalization effect and the examination of youth slang in multimedia learning , 2015, Comput. Hum. Behav..

[16]  Raphael Meyer von Wolff,et al.  Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents , 2019, Wirtschaftsinformatik.

[17]  Emiel Krahmer,et al.  The Effects of Feedback on Children's Engagement and Learning Outcomes in Robot-Assisted Second Language Learning , 2020, Frontiers in Robotics and AI.

[18]  R. Mayer,et al.  Getting the point: Which kinds of gestures by pedagogical agents improve multimedia learning? , 2019, Journal of Educational Psychology.

[19]  Jason D. Moss,et al.  The mediating role of presence differs across types of spatial learning in immersive technologies , 2020, Comput. Hum. Behav..

[20]  Flemming Konradsen,et al.  A review of the use of virtual reality head-mounted displays in education and training , 2017, Education and Information Technologies.

[21]  Ausdang Thangthai,et al.  Synthesising visual speech using dynamic visemes and deep learning architectures , 2019, Comput. Speech Lang..

[22]  Stefan Kopp,et al.  Second Language Tutoring Using Social Robots: A Large-Scale Study , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[23]  Kyle Johnsen,et al.  Experiences in using immersive virtual characters to educate medical communication skills , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[24]  James C. Lester,et al.  The Case for Social Agency in Computer-Based Teaching: Do Students Learn More Deeply When They Interact With Animated Pedagogical Agents? , 2001 .

[25]  Jeeheon Ryu,et al.  The effects of social and cognitive cues on learning comprehension, eye-gaze pattern, and cognitive load in video instruction , 2020, Journal of Computing in Higher Education.

[26]  Yu-Yun Chang,et al.  Evaluation of TTS Systems in Intelligibility and Comprehension Tasks , 2011, ROCLING/IJCLCLP.

[27]  S A Duffy,et al.  Comprehension of Synthetic Speech Produced by Rule: A Review and Theoretical Interpretation , 1992, Language and speech.

[28]  Richard E. Mayer,et al.  Multimedia learning in an interactive self-explaining environment: What works in the design of agent , 2003 .

[29]  Annika Silvervarg,et al.  Instructing a Teachable Agent with Low or High Self-Efficacy – Does Similarity Attract? , 2019, International Journal of Artificial Intelligence in Education.

[30]  M A Crary,et al.  Dynamic magnetic resonance imaging in the study of vocal tract configuration. , 1996, Journal of voice : official journal of the Voice Foundation.

[31]  L. Bernstein,et al.  Enhanced visual speech perception in individuals with early-onset hearing impairment. , 2007, Journal of speech, language, and hearing research : JSLHR.

[32]  Catherine J. Stevens,et al.  On-line experimental methods to evaluate text-to-speech (TTS) synthesis: effects of voice gender and signal quality on intelligibility, naturalness and preference , 2005, Comput. Speech Lang..

[33]  Deborah Richards,et al.  Supporting and challenging learners through pedagogical agents: Addressing ethical issues through designing for values , 2019, Br. J. Educ. Technol..

[34]  R. Mayer,et al.  Fostering social agency in multimedia learning: Examining the impact of an animated agent’s voice ☆ , 2005 .

[35]  Tze Wei Liew,et al.  The Effects of Peer-Like and Expert-Like Pedagogical Agents on Learners' Agent Perceptions, Task-Related Attitudes, and Learning Achievement , 2013, J. Educ. Technol. Soc..

[36]  Lynne E. Bernstein,et al.  Visual speech discrimination and identification of natural and synthetic consonant stimuli , 2015, Front. Psychol..

[37]  Marian L. Houser,et al.  The teacher‐student relationship as an interpersonal relationship , 2000 .

[38]  P. Arnold,et al.  Bisensory augmentation: a speechreading advantage when speech is clearly audible and intact. , 2001, British journal of psychology.

[39]  Ginevra Castellano,et al.  Empathic Robot for Group Learning , 2019, ACM Transactions on Human-Robot Interaction.

[40]  Christine L. Lisetti,et al.  10 advantages of using avatars in patient-centered computer-based interventions for behavior change , 2012, SIGH.

[41]  Paul Brna,et al.  Perfect presence: What does this mean for the design of virtual learning environments? , 2000, Education and Information Technologies.

[42]  D. Kort,et al.  Digital Games as Social Presence Technology: Development of the Social Presence in Gaming Questionnaire (SPGQ) , 2007 .

[43]  Maik Beege,et al.  Boundary conditions of the politeness effect in online mathematical learning , 2019, Comput. Hum. Behav..

[44]  Guan-Yu Lin,et al.  Social Presence Questionnaire of Online Collaborative Learning: Development and Validity , 2005 .

[45]  Tze Wei Liew,et al.  Does speaker’s voice enthusiasm affect social cue, cognitive load and transfer in multimedia learning? , 2020, Information and Learning Sciences.

[46]  Stefan Kopp,et al.  The Effect of a Robot’s Gestures and Adaptive Tutoring on Children’s Acquisition of Second Language Vocabularies , 2018, 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[47]  P. Arnold,et al.  Bisensory augmentation of complex spoken passages , 2001, British journal of audiology.

[48]  Yanghee Kim,et al.  Research-Based Design of Pedagogical Agent Roles: a Review, Progress, and Recommendations , 2016, International Journal of Artificial Intelligence in Education.

[49]  James A. Landay,et al.  QuizBot: A Dialogue-based Adaptive Learning System for Factual Knowledge , 2019, CHI.

[50]  Baptiste Jacquet,et al.  Learning by Teaching with Humanoid Robot: A New Powerful Experimental Tool to Improve Children's Learning Ability , 2018, J. Robotics.

[51]  R. Mayer,et al.  Personalized messages that promote science learning in virtual environments , 2004 .

[52]  Nicholas H. Müller,et al.  Cognitive Load Levels While Learning with or Without a Pedagogical Agent , 2019, HCI.

[53]  Roxana Moreno,et al.  The Roles of Animated Pedagogical Agents' Presence and Nonverbal Communication in Multimedia Learning Environments , 2010, J. Media Psychol. Theor. Methods Appl..

[54]  Patricia D. Mautone,et al.  Social cues in multimedia learning: Role of speaker's voice. , 2003 .

[55]  Nor Azan Mat Zin,et al.  Exploring the affective, motivational and cognitive effects of pedagogical agent enthusiasm in a multimedia learning environment , 2017, Human-centric Computing and Information Sciences.

[56]  Frank E. Ritter,et al.  Enhanced Behavioral Realism for Live Fire Targets , 2014 .

[57]  Robert O. Davis The impact of pedagogical agent gesturing in multimedia learning environments: A meta-analysis , 2018, Educational Research Review.

[58]  Noah L. Schroeder,et al.  How Effective are Pedagogical Agents for Learning? A Meta-Analytic Review , 2013 .

[59]  J. Gratch,et al.  Who Is there? Can a Virtual Agent Really Elicit Social Presence? , 2009 .

[60]  Mohd Hishamuddin Abdul Rahman,et al.  Sense of Presence and Learning Satisfaction among Students of Different Age Groups in a 3-D Virtual World , 2019, International Journal on Advanced Science, Engineering and Information Technology.

[61]  Noah L. Schroeder,et al.  Reconsidering the voice effect when learning from a virtual human , 2017, Comput. Educ..

[62]  Adriana Tapus,et al.  User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy , 2008, Intell. Serv. Robotics.

[63]  James C. Lester,et al.  The persona effect: affective impact of animated pedagogical agents , 1997, CHI.

[64]  Robert O. Davis,et al.  Sometimes more is better: Agent gestures, procedural knowledge and the foreign language learner , 2019, Br. J. Educ. Technol..

[65]  Wendy Ju,et al.  Social robots and virtual agents as lecturers for video instruction , 2016, Comput. Hum. Behav..

[66]  C. Nass,et al.  Social-Psychological Origins of Feelings of Presence: Creating Social Presence With Machine-Generated Voices , 2005 .

[67]  S. Trauzettel-Klosinski,et al.  Standardized assessment of reading performance: the New International Reading Speed Texts IReST. , 2012, Investigative ophthalmology & visual science.

[68]  G. Bente,et al.  Personalizing e-Learning. The Social Effects of Pedagogical Agents , 2010 .

[69]  David Wood,et al.  The effect of task conditions on the comprehensibility of synthetic speech , 2000, CHI.

[70]  Jerald Thomas,et al.  Level of immersion affects spatial learning in virtual environments: results of a three-condition within-subjects study with long intersession intervals , 2020, Virtual Reality.

[71]  Soo Youn Oh,et al.  Effect of Behavioral Realism on Social Interactions Inside Collaborative Virtual Environments , 2018, PRESENCE: Virtual and Augmented Reality.

[72]  Carrie Heeter,et al.  Being There: The Subjective Experience of Presence , 1992, Presence: Teleoperators & Virtual Environments.

[73]  J. Bailenson,et al.  Virtual Humans and Persuasion: The Effects of Agency and Behavioral Realism , 2007 .

[74]  Joris B. Janssen,et al.  Using a robot to personalise health education for children with diabetes type 1: a pilot study. , 2013, Patient education and counseling.

[75]  Zehui Zhan,et al.  Academic self-concept and social presence in face-to-face and online learning: Perceptions and effects on students' learning achievement and satisfaction across environments , 2013, Comput. Educ..

[76]  Hartmut Traunmüller,et al.  Perception of vocal effort and distance from the speaker on the basis of vowel utterances , 2002, Perception & psychophysics.

[77]  A. Macleod,et al.  Quantifying the contribution of vision to speech perception in noise. , 1987, British journal of audiology.

[78]  Jeremy N. Bailenson,et al.  How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence , 2016 .

[79]  Neil Charness,et al.  Age Differences in Identifying Words in Synthetic Speech , 2007, Hum. Factors.

[80]  Mark G. Core,et al.  Learning intercultural communication skills with virtual humans: Feedback and fidelity. , 2013 .

[81]  Wesley Mattheyses,et al.  Audiovisual speech synthesis: An overview of the state-of-the-art , 2015, Speech Commun..

[82]  R. Mayer,et al.  Engaging students in active learning: The case for personalized multimedia messages. , 2000 .

[83]  Simon King,et al.  Multisyn: Open-domain unit selection for the Festival speech synthesis system , 2007, Speech Commun..

[84]  Nadia Magnenat-Thalmann,et al.  Building long-term relationships with virtual and robotic characters: the role of remembering , 2011, The Visual Computer.

[85]  Tae Jung Park,et al.  Reconsidering the Voice Prinicple with Non-native Language Speakers , 2019, Comput. Educ..

[86]  Patrick Gebhard,et al.  Exploring interaction strategies for virtual characters to induce stress in simulated job interviews , 2014, AAMAS.

[87]  Amy Ogan,et al.  "Oh dear stacy!": social interaction, elaboration, and learning with teachable agents , 2012, CHI.

[88]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[89]  Steve Nebel,et al.  Mind your Ps and Qs! How polite instructions affect learning with multimedia , 2015, Comput. Hum. Behav..

[90]  R. Mayer,et al.  An embodiment effect in computer-based learning with animated pedagogical agents. , 2012, Journal of experimental psychology. Applied.

[91]  C. Heyes Born Pupils? Natural Pedagogy and Cultural Pedagogy , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[92]  Ning Wang,et al.  The politeness effect: Pedagogical agents and learning outcomes , 2008, Int. J. Hum. Comput. Stud..

[93]  Noah L. Schroeder,et al.  Text-to-Speech Software and Learning: Investigating the Relevancy of the Voice Effect , 2018, Journal of Educational Computing Research.

[94]  Richard D. Gilson,et al.  Linguistic Cues and Memory for Synthetic and Natural Speech , 2000, Hum. Factors.

[95]  Harry Budi Santoso,et al.  The Design and Impact of the Pedagogical Agent: A Systematic Literature Review , 2019, The Journal of Educators Online.

[96]  W. H. Sumby,et al.  Visual contribution to speech intelligibility in noise , 1954 .

[97]  Wafa Johal,et al.  When deictic gestures in a robot can harm child-robot collaboration , 2018, IDC.

[98]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .