Nonverbally Smart User Interfaces: Postural and Facial Expression Data in Human Computer Interaction

We suggest that User Interfaces (UIs) can be designed to serve as cognitive tools based on a model of nonverbal human interaction. Smart User Interfaces (SUIs) have the potential to support the human user when and where appropriate and thus indirectly facilitate higher mental processes without the need for end-user programming or external actuation. Moreover, graphic nonverbally sensitive SUIs are expected to be less likely to interfere with ongoing activity and disrupt the user. We present two non-invasive methods to assess postural and facial expression components and propose a contextual analysis to guide SUI actuation and supportive action. The approach is illustrated in a possible redesign of the Microsoft helper agent "Clippit" ®.

[1]  Keith Waters,et al.  Computer facial animation , 1996 .

[2]  Randolph R. Cornelius,et al.  The science of emotion: Research and tradition in the psychology of emotion. , 1997 .

[3]  P. Ekman Emotion in the human face , 1982 .

[4]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[5]  Jörgen Ahlberg Model-based coding : extraction, coding, and evaluation of face model parameters , 2002 .

[6]  C. Darwin The Expression of Emotion in Man and Animals , 2020 .

[7]  Jonathan Klein,et al.  This computer responds to user frustration , 1999, CHI Extended Abstracts.

[8]  P. Ekman,et al.  Facial Expressions of Emotion , 1979 .

[9]  Igor S. Pandzic,et al.  MPEG-4 Facial Animation , 2002 .

[10]  Roy Stripling,et al.  Postural Control as a Probe for Cognitive State: Exploiting Human Information Processing to Enhance Performance , 2004, Int. J. Hum. Comput. Interact..

[11]  W. Lewis Johnson,et al.  Pedagogical agents on the Web , 1999, AGENTS '99.

[12]  G. Fraenkel,et al.  The Orientation of Animals, Kineses, Taxes and Compass Reactions, , 1941 .

[13]  Jonathan Klein,et al.  This computer responds to user frustration: Theory, design, and results , 2002, Interact. Comput..

[14]  Piet Kommers,et al.  Cognitive Tools for Learning , 2001, NATO ASI Series.

[15]  K. Deaux,et al.  Nonverbal Behavior and Communication (2nd ed.). , 1987 .

[16]  Jeffrey M. Zacks,et al.  Event structure in perception and conception. , 2001, Psychological bulletin.

[17]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[18]  David H. Jonassen,et al.  What are Cognitive Tools , 1992 .