The Role of Gesture in Document Explanation by Embodied Conversational Agents

We describe two empirical studies of how professionals explain documents to lay clients who have varying levels of knowledge about the domain under discussion. We find that hand gestures, and in particular deictic gestures by the professional at various parts of the document play a major role in explanations of documents with clients in face-to-face settings. We describe a preliminary computational model of document explanation by an embodied conversational agent, in which appropriate form and location of hand gestures are used by the agent in explaining a document to a user. Results from a pilot evaluation study indicate that individuals with low levels of domain knowledge prefer receiving explanations from such an agent rather than from a human. Examples are drawn from the healthcare domain, in which research consent forms and hospital discharge instruction forms are used as the documents being explained, and health literacy is used as the measure of client domain knowledge.

[1]  Stefan Kopp,et al.  Towards a Common Framework for Multimodal Generation: The Behavior Markup Language , 2006, IVA.

[2]  Hao Yan,et al.  Shared reality: physical collaboration with a virtual peer , 2000, CHI Extended Abstracts.

[3]  S. Eckstein.  The Belmont Report: ethical principles and guidelines for the protection of human subjects of research , 2003 .

[4]  Arthur C. Graesser,et al.  AutoTutor: A simulation of a human tutor , 1999, Cognitive Systems Research.

[5]  Yukiko I. Nakano,et al.  MACK: Media lab Autonomous Conversational Kiosk , 2002 .

[6]  Henry K. Beecher.  Ethics and Clinical Research , 2001 .

[7]  Michael E. J. Masson,et al.  Comprehension of legal contracts by non‐experts: Effectiveness of plain language redrafting , 1994 .

[8]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[9]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[10]  Arthur D. Fisk,et al.  Human Factors Interventions for the Health Care of Older Adults , 2008 .

[11]  E. J. Mayeaux,et al.  Rapid estimate of adult literacy in medicine: a shortened screening instrument. , 1993, Family medicine.

[12]  Robert Dale,et al.  Building Natural Language Generation Systems: Figures , 2000 .

[13]  L. G. Doak,et al.  Teaching Patients With Low Literacy Skills , 1985 .

[14]  D. Traum,et al.  Coding Discourse Structure in Dialogue (Version 1.0) , 1999 .

[15]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[16]  S. Kita.  Pointing: Where language, culture, and cognition meet , 2003 .

[17]  Ipke Wachsmuth,et al.  Incremental Generation of Multimodal Deixis Referring to Objects , 2005, ENLG.

[18]  Michael K Paasche-Orlow,et al.  Readability standards for informed-consent forms as compared with actual readability. , 2003, The New England journal of medicine.

[19]  Yukiko I. Nakano,et al.  Non-Verbal Cues for Discourse Structure , 2022 .

[20]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[21]  Yukiko I. Nakano,et al.  Towards a Model of Face-to-Face Grounding , 2003, ACL.

[22]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[23]  Sarah M. Greene,et al.  How health care systems can begin to address the challenge of limited literacy , 2006, Journal of General Internal Medicine.

[24]  William D. Marslen-Wilson,et al.  Discourse: The Establishment and Maintenance of Reference , 2022 .

[25]  David P. Wilkins.  Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). , 2003 .

[26]  J. Cassell,et al.  Modeling Gaze Behavior as a Function of Discourse Structure , 1998 .

[27]  W. Lewis Johnson,et al.  Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control , 1999, Appl. Artif. Intell..

[28]  Timothy W. Bickmore,et al.  Establishing and maintaining long-term human-computer relationships , 2005, TCHI.

[29]  Penelope Sibun.  GENERATING TEXT WITHOUT TREES , 1992 .

[30]  Robert Edson,et al.  Evaluating the quality of informed consent , 2005, Clinical trials.

[31]  Stephanie D. Teasley,et al.  Perspectives on socially shared cognition , 1991 .

[32]  James F. Allen,et al.  Draft of DAMSL Dialog Act Markup in Several Layers , 2007 .

[33]  Mitchell P. Marcus,et al.  Form: an experiment in the annotation of the kinetics of gesture , 2005 .

[34]  Timothy W. Bickmore,et al.  Health Document Explanation by Virtual Agents , 2007, IVA.

[35]  Michael S. Wogalter.  On the adequacy of legal documents: factors that influence informed consent , 1999 .

[36]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[37]  James C. Lester,et al.  Life-Like Pedagogical Agents in Constructivist Multimedia Environments: Cognitive Consequences of their Interaction , 2000 .

[38]  Norman I. Badler,et al.  Design of a Virtual Human Presenter , 2000, IEEE Computer Graphics and Applications.

[39]  J. Ogloff,et al.  Are research participants truly informed? Readability of informed consent forms used in research. , 1991, Ethics & behavior.

[40]  Darren A. DeWalt,et al.  Literacy and health outcomes , 2006, Journal of General Internal Medicine.

[41]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[42]  Thomas Rist,et al.  Integrating reactive and scripted behaviors in a life-like presentation agent , 1998, AGENTS '98.

[43]  F Davidoff,et al.  Databases in the Next Millennium , 1997, Annals of Internal Medicine.

[44]  Charles B. Callaway,et al.  Cosmo: A Life-like Animated Pedagogical Agent with Deictic Believability (cid:3) , 1997 .

[45]  T. Bickmore.  Relational agents : effecting change through human-computer relationships , 2003 .

[46]  L A Morris,et al.  Effects of written drug information on patient knowledge and compliance: a literature review. , 1979, American journal of public health.