Speech and Gesture Interaction in an Ambient Assisted Living Lab

In this paper we describe our recent and future research on multimodal interaction in an Ambient Assisted Living Lab. Our work combines two interaction modes, speech and gesture, for multiple device control in Ambient Assisted Living environments. We conducted a user study concerning multimodal interaction between participants and an intelligent wheelchair in a smart home environment. Important empirical data were collected through the user study, which encouraged further developments on our multimodal interactive system for Ambient Assisted Living environments.

[1]  Sotaro Kita,et al.  How representational gestures help speaking , 2000 .

[2]  Staffan Larsson,et al.  Information state and dialogue management in the TRINDI dialogue move engine toolkit , 2000, Natural Language Engineering.

[3]  R. Krauss,et al.  PSYCHOLOGICAL SCIENCE Research Article GESTURE, SPEECH, AND LEXICAL ACCESS: The Role of Lexical Movements in Speech Production , 2022 .

[4]  Elisabeth André,et al.  Wave like an Egyptian: accelerometer based gesture recognition for culture specific interactions , 2008, BCS HCI.

[5]  Barbara Di Eugenio,et al.  Improving Pronominal and Deictic Co-Reference Resolution with Multi-Modal Features , 2011, SIGDIAL Conference.

[6]  Eric Becker,et al.  Event-based experiments in an assistive environment using wireless sensor networks and voice recognition , 2009, PETRA '09.

[7]  Kawa Nazemi,et al.  Natural Gesture Interaction with Accelerometer-Based Devices in Ambient Assisted Environments , 2011 .

[8]  M. Halliday,et al.  AN INTRODUCTION TO FUNCTIONAL GRAMMAR (Third Edition) , 2022 .

[9]  Cui Jian,et al.  Evaluating a Spoken Language Interface of a Multimodal Interactive Guidance System for Elderly Persons , 2012, HEALTHINF.

[10]  Bernard Rimé,et al.  Fundamentals of nonverbal behavior , 1991 .

[11]  Hannes Rieser,et al.  Explaining Speech Gesture Alignment in MM Dialogue Using Gesture Typology , 2010 .

[12]  Michael Strube,et al.  A Machine Learning Approach to Pronoun Resolution in Spoken Dialogue , 2003, ACL.

[13]  Margot Mieskes,et al.  An Intelligent House Control Using Speech Recognition with Integrated Localization , 2011 .

[14]  A. Leroi‐Gourhan,et al.  Gesture and Speech , 1993 .

[15]  Verena Fuchsberger,et al.  Ambient assisted living: elderly people's needs and how to face them , 2008, SAME '08.

[16]  Ricardo Costa,et al.  Ambient Assisted Living , 2009 .

[17]  John A. Bateman,et al.  Daisie: Information State Dialogues for Situated Systems , 2009, TSD.

[18]  Stefan Kopp.  The spatial specificity of iconic gestures , 2005 .

[19]  Nicole Chovil.  Discourse‐oriented facial displays in conversation , 1991 .

[20]  Shinya Takahashi,et al.  Dialogue Experiment for Elderly People in Home Health Care System , 2003, TSD.

[21]  Jarek Krajewski,et al.  An Acoustic Framework for Detecting Fatigue in Speech Based Human-Computer-Interaction , 2008, ICCHP.

[22]  Niko Moritz,et al.  Acoustic user interfaces for ambient-assisted living technologies , 2010, Informatics for health & social care.

[23]  S. Goldin-Meadow,et al.  Hearing Gesture: How Our Hands Help Us Think , 2003 .

[24]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[25]  Martha W. Alibali,et al.  Raise your hand if you’re spatial: Relations between verbal and spatial skills and gesture production , 2007 .

[26]  James F. Allen,et al.  Resolving Demonstrative Anaphora in the TRAINS93 Corpus , 1998 .

[27]  Jan Alexandersson,et al.  A Gesture Based System for Context – Sensitive Interaction with Smart Homes , 2011 .

[28]  Michael Neff,et al.  An annotation scheme for conversational gestures: how to economically capture timing and form , 2007, Lang. Resour. Evaluation.

[29]  Martha W. Alibali,et al.  Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information , 2005, Spatial Cogn. Comput..

[30]  Thora Tenbrink,et al.  A linguistic ontology of space for natural language processing , 2010, Artif. Intell..

[31]  Daniel Couto Vale,et al.  Using Foot-Syllable Grammars to Customize Speech Recognizers for Dialogue Systems , 2012, TSD.

[32]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[33]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[34]  Sotaro Kita,et al.  Cross-cultural variation of speech-accompanying gesture: A review , 2009, Speech Accompanying-Gesture.

[35]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[36]  Michael Halliday,et al.  An Introduction to Functional Grammar , 1985 .

[37]  Jacob Eisenstein,et al.  Gesture Improves Coreference Resolution , 2006, HLT-NAACL.