Five Design Principles for Experiments on the Effects of Animated Pedagogical Agents

Research on animated pedagogical agents (agents) is viewed as a very positive attempt to introduce more pedagogical support and motivational elements into multi-media instruction. Yet, existing empirical studies that examine the learning benefits of agents have had very mixed results, largely due to the way that they are designed. This article will suggest five design principles for future research on the impact of agents on learning and motivation including: 1) The Balanced Separation Principle describes need for adequate controls that tease out the specific type of learning and/or motivational support the agent is providing; 2) The Variety of Outcomes Principle suggests different measures to test a variety of learning and motivation outcomes that may be influenced by agents; 3) The Robust Measurement Principle advises researchers to pay special attention to the reliability and construct validity of experimenter designed measures; 4) The Cost-Effectiveness Principle recommends the collection of data on the relative cost of producing agent and non-agent treatments; and 5) The Cognitive Load Principle alerts those who plan treatments for experiments to exercise caution when developing and testing agents that are visually and aurally “noisy” or complex.

[1]  Madhabi Chatterji,et al.  Evidence on “What Works”: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs , 2004 .

[2]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[3]  Gary R. Morrison,et al.  In search of a happy medium in instructional technology research: Issues concerning external validity, media replications, and learner control , 1989 .

[4]  Gordon I. McCalla,et al.  The Caring Personal Agent , 2003, Int. J. Artif. Intell. Educ..

[5]  Richard E. Clark,et al.  Learning from media : arguments, analysis, and evidence , 2001 .

[6]  N. Gage,et al.  Second handbook of research on teaching : a project of the American Educational Research Association , 1973 .

[7]  Jeeheon Ryu,et al.  The Effects of Image and Animation in Enhancing Pedagogical Agent Persona , 2003 .

[8]  Scotty D. Craig,et al.  Constructing Knowledge from Dialog in an Intelligent Tutoring System: Interactive Learning, Vicarious Learning, and Pedagogical Agents , 2004 .

[9]  James C. Lester,et al.  Animated pedagogical agents and problem-solving effectiveness: a large-scale empirical evaluation , 1997 .

[10]  Ralph E. Reynolds,et al.  Effect of Interest on Attention and Learning , 1988 .

[11]  K. Scherer,et al.  How Seductive Details Do Their Damage : A Theory of Cognitive Interest in Science Learning , 2004 .

[12]  H. Levin,et al.  Cost-Effectiveness Analysis: Methods and Applications , 2000 .

[13]  D. Massaro,et al.  Development and Evaluation of a Computer-Animated Tutor for Vocabulary and Language Learning in Children with Autism , 2003, Journal of autism and developmental disorders.

[14]  T. Koda,et al.  Agents with faces: the effect of personification , 1996, Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN'96 TSUKUBA.

[15]  Scotty D. Craig,et al.  Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy , 2002 .

[16]  Slava Kalyuga,et al.  Managing split-attention and redundancy in multimedia instruction , 1999 .

[17]  Thomas Rist,et al.  WebPersona: a lifelike presentation agent for the World-Wide Web , 1998, Knowl. Based Syst..

[18]  Human Factors in Computing Systems, CHI '95 Conference Proceedings, Denver, Colorado, USA, May 7-11, 1995 , 1995, CHI.

[19]  Ellen J. Langer,et al.  The illusion of calculated decisions. , 1994 .

[20]  Lisa M. PytlikZillig,et al.  Web Based Learning: What do we know? Where do we go? , 2003 .

[21]  Akikazu Takeuchi,et al.  Situated facial displays: towards social interaction , 1995, CHI '95.

[22]  Hiroaki Ogata,et al.  Neclle: Network-based communicative language-learning environment focusing on communicative gaps , 2001, Comput. Educ..

[23]  Justine Cassell,et al.  Virtual peers as partners in storytelling and literacy learning , 2003, J. Comput. Assist. Learn..

[24]  Alfred Bork,et al.  Multimedia in Learning , 2001 .

[25]  Arthur C. Graesser,et al.  A framework of synthesizing tutoring conversation capability with web-based distance education courseware , 2004, Comput. Educ..

[26]  Richard E. Clark,et al.  Evaluating Distance Education: Strategies and Cautions , 2000 .

[27]  Slava Kalyuga,et al.  Measuring Knowledge to Optimize Cognitive Load Factors During Instruction. , 2004 .

[28]  C. Nass,et al.  Voices, boxes, and sources of messages: Computers and social actors. , 1993 .

[29]  R. Atkinson.  Optimizing learning from examples using animated pedagogical agents. , 2002 .

[30]  Mark K. Singley,et al.  Deploying Intelligent Tutors on the Web: An Architecture and an Example , 1999 .

[31]  Henry M. Levin,et al.  Cost-Effectiveness of Computer-Assisted Instruction , 1987 .

[32]  Antonija Mitrovic,et al.  Evaluating an Animated Pedagogical Agent , 2000, Intelligent Tutoring Systems.

[33]  R. Mayer,et al.  Multimedia Learning: The Promise of Multimedia Learning , 2001 .

[34]  Clifford Nass,et al.  Anthropomorphism, agency, and ethopoeia: computers as social actors , 1993, INTERCHI Adjunct Proceedings.

[35]  Demetrios G. Sampson,et al.  Personalised Learning: Educational, Technological and Standardisation Perspective , 2002 .

[36]  Richard E. Clark,et al.  The Development of Authentic Educational Technologies. , 1999 .

[37]  Susanne van Mulken,et al.  The impact of animated interface agents: a review of empirical research , 2000, Int. J. Hum. Comput. Stud..

[38]  James C. Lester,et al.  The Case for Social Agency in Computer-Based Teaching: Do Students Learn More Deeply When They Interact With Animated Pedagogical Agents? , 2001 .

[39]  James C. Lester,et al.  Achieving Affective Impact: Visual Emotive Communication in Lifelike Pedagogical Agents , 1999 .

[40]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[41]  Richard E. Clark,et al.  A Proposal for the Collaborative Development of Authentic Performance Technology. , 2000 .

[42]  Thomas Erickson.  Designing agents as if people mattered , 1997 .

[43]  Steve Hackbarth,et al.  Introduction to special issue on Web-based learning , 1997 .

[44]  F. Paas,et al.  Measurement of Cognitive Load in Instructional Research , 1994, Perceptual and motor skills.

[45]  Lee Sproull,et al.  When the Interface Is a Face , 1996, Hum. Comput. Interact..

[46]  Amy L. Baylor,et al.  Agent-Based Learning Environments as a Research Tool for Investigating Teaching and Learning , 2002 .

[47]  R. Mayer,et al.  The Role of Interest in Learning From Scientific Text and Illustrations: On the Distinction Between Emotional Interest and Cognitive Interest , 1997 .

[48]  Richard C. Anderson.  Interestingness of children's reading material , 1984 .

[49]  Maria Virvou,et al.  Evaluating the persona effect of an interface agent in a tutoring system , 2002, J. Comput. Assist. Learn..

[50]  P. Chandler,et al.  Why Some Material Is Difficult to Learn , 1994 .

[51]  Thomas Rist,et al.  Employing AI Methods to Control the Behavior of Animated Interface Agents , 1999, Appl. Artif. Intell..

[52]  Roger C. Schank,et al.  Beliefs, Reasoning and Decision-making: Psycho-logic in Honor of Bob Abelson , 1994 .

[53]  Lisa M. PytlikZillig,et al.  Web-Based Learning: What Do We Know? Where Do We Go? Nebraska Symposium on Information Technology in Education (1st, Lincoln, Nebraska, May 15-17, 2002). , 2003 .

[54]  L. Fielding.  INTERESTINGNESS OF CHILDREN'S READING MATERIAL , 2007 .

[55]  Jeroen J. G. van Merriënboer,et al.  Training Complex Cognitive Skills: A Four-Component Instructional Design Model for Technical Training , 1997 .

[56]  Richard E. Mayer,et al.  Multimedia learning in an interactive self-explaining environment: What works in the design of agent , 2003 .

[57]  James C. Lester,et al.  Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments , 2000 .

[58]  Tom Cobb,et al.  Cognitive efficiency: Toward a revised theory of media , 1997 .

[59]  Elisabeth André,et al.  The Persona Effect: How Substantial Is It? , 1998, BCS HCI.

[60]  Arthur C. Graesser,et al.  Simulating Human Tutor Dialog Moves in AutoTutor , 2001 .

[61]  Richard E. Clark,et al.  Yin and Yang Cognitive Motivational Processes Operating in Multimedia Learning Environments , 2004 .

[62]  A. L. Baylor.  Expanding preservice teachers' metacognitive awareness of instructional planning through pedagogical agents , 2002 .

[63]  F. Paas.  Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. , 1992 .

[64]  Gary R. Morrison,et al.  EXPERIMENTAL RESEARCH METHODS , 2003 .

[65]  Kristinn R. Thórisson,et al.  The Power of a Nod and a Glance: Envelope Vs. Emotional Feedback in Animated Conversational Agents , 1999, Appl. Artif. Intell..