Design of Multimodal Interaction with Mobile Devices - Challenges for Visually Impaired and Elderly Users

This paper presents two early studies aimed at investigating issues concerning the design of multimodal interaction based on voice commands and mid-air gestures with mobile technology specifically designed for visually impaired and elderly users. These studies have been carried out on a new device allowing enhanced speech recognition (interpreting lip movements) and mid-air gesture interaction on Android devices (smartphone and tablet PC). The initial findings and challenges raised by these novel interaction modalities are discussed. These mainly centre on issues of feedback and feedforward, the avoidance of false positives and point of reference or orientation issues regarding the device and the mid-air gestures.

[1]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[2]  Stephen Lindsay,et al.  Engaging older people using participatory design , 2012, CHI.

[3]  Elise van den Hoven,et al.  Crossing the bridge over Norman's Gulf of Execution: revealing feedforward's true identity , 2013, CHI.

[4]  Peter Gregor,et al.  Designing for dynamic diversity: interfaces for older people , 2002, Assets '02.

[5]  Olivier Bau,et al.  OctoPocus: a dynamic guide for learning gesture-based command sets , 2008, UIST '08.

[6]  Sukeshini A. Grandhi,et al.  Understanding naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces , 2011, CHI.

[7]  Philip Kortum,et al.  HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces , 2008 .

[8]  Patrick Langdon,et al.  Physical gestures for abstract concepts: Inclusive design with primary metaphors , 2010, Interact. Comput..

[9]  Xiang 'Anthony' Chen,et al.  Air+touch: interweaving touch & in-air gestures , 2014, UIST.

[10]  E B Ryan,et al.  Attitudes toward younger and older adults learning to use computers. , 1992, Journal of gerontology.

[11]  Dennis Wixon,et al.  Touch versus In-Air Gestures , 2011 .

[12]  Heiko Horst Hornung,et al.  Gesture-Based Interaction in Domotic Environments: State of the Art and HCI Framework Inspired by the Diversity , 2013, INTERACT.

[13]  Uran Oh,et al.  Audio-Based Feedback Techniques for Teaching Touchscreen Gestures , 2015, ACM Trans. Access. Comput..

[14]  Arthur D. Fisk,et al.  Designing for Older Adults: Principles and Creative Human Factors Approaches , 2004 .

[15]  Stephan Wensveen,et al.  Interaction frogger: a design framework to couple action and function through feedback and feedforward , 2004, DIS '04.