Mixed Reality Methods. Preliminary Considerations for Multimodal Analysis of Human-Agent Interactions

The ongoing development of embodied conversational agents requires a precise analysis of human-agent interaction. Currently, however, there are still only few approaches that investigate interactions by means of multimodal methods and both the individual reflection of experience and the interactive behavior. In this paper, we present a methodological approach that allows collecting data on individual perceptions of interacting with virtual agents as well as on the interaction itself. By means of mixed reality, the jointly coordinated behavior of users and agents in virtual spaces can be captured. This approach enables a more comprehensive understanding of the complex dynamics of human-agent interactions and offers the advantage of combining different types of data.

[1]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[2]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[3]  J. Chatwin.  Conversation analysis. , 2004, Complementary therapies in medicine.

[4]  Hung-Hsuan Huang,et al.  Embodied Conversational Agents , 2009 .

[5]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.

[6]  Mark Levine,et al.  Overcoming the uncanny valley: Displays of emotions reduce the uncanniness of humanlike robots , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[7]  A. Gillespie,et al.  A truly human interface: interacting face-to-face with someone whose words are determined by a computer program , 2015, Front. Psychol..

[8]  Quoc V. Le,et al.  Towards a Human-like Open-Domain Chatbot , 2020, ArXiv.

[9]  Sebastian Möller,et al.  Evaluating embodied conversational agents in multimodal interfaces , 2015 .

[10]  Stefan Kopp,et al.  A personality-based emotional model for embodied conversational agents: Effects on perceived social presence and game experience of users , 2019, Entertain. Comput..

[11]  Benjamin R. Cowan,et al.  Mapping Perceptions of Humanness in Intelligent Personal Assistant Interaction , 2019, MobileHCI.

[12]  Li Gong,et al.  How social is social responses to computers? The function of the degree of anthropomorphism in computer representations , 2008, Comput. Hum. Behav..

[13]  H. Knoblauch,et al.  Videography: An Interpretative Approach to Video-Recorded Micro-Social Interaction , 2020, The SAGE Handbook of Visual Research Methods.

[14]  C. Nass,et al.  Machines and Mindlessness , 2000 .

[15]  Jens Reinhardt,et al.  Embedding Conversational Agents into AR: Invisible or with a Realistic Human Body? , 2020, TEI.

[16]  Deborah Richards,et al.  What are We Measuring Anyway?: - A Literature Survey of Questionnaires Used in Studies Reported in the Intelligent Virtual Agent Conferences , 2019, BNAIC/BENELEARN.

[17]  Raymond Chiong,et al.  Avatars and Embodied Agents in Experimental Information Systems Research: A Systematic Review and Conceptual Framework , 2019, Australas. J. Inf. Syst..

[18]  Jeremy N. Bailenson,et al.  A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces , 2007, CHI.

[19]  Torsten W. Kuhlen,et al.  Evaluating the Influence of Phoneme-Dependent Dynamic Speaker Directivity of Embodied Conversational Agents' Speech , 2020, IVA.

[20]  Isaac Wang,et al.  Exploring Virtual Agents for Augmented Reality , 2019, CHI.

[21]  Mike Seymour,et al.  Meet Mike: epic avatars , 2017, SIGGRAPH VR Village.

[22]  Sven Strauss,et al.  Pragmatics Of Human Communication , 2016 .

[23]  Frank Schneider,et al.  Multimodal human communication — Targeting facial expressions, speech content and prosody , 2012, NeuroImage.

[24]  Mary Ellen Foster.  Face-to-face conversation: why embodiment matters for conversational user interfaces , 2019, CUI.

[25]  Willem-Paul Brinkman,et al.  The 19 Unifying Questionnaire Constructs of Artificial Social Agents: An IVA Community Analysis , 2020, IVA.