User Interaction in Mobile Navigation Applications

The chapter focuses on cooperation and interaction in multimodal route navigation and provides an overview of the advantages and disadvantages of multimodal interaction in location-based services in general. The goal of the research has been to study methods and techniques for richer human-computer interaction, and to investigate interconnection and user preferences concerning speech and tactile input modalities on a route navigation task. The chapter also surveys the work on a mobile navigation application which allows the user to query public transportation routes using speech and pen pointing gestures. The first version of the PDA-based navigation application, MUMS, has been developed with the Helsinki City public transportation as the domain, and the user can ask timetable and navigation information either by natural language questions or clicking on the map. On the basis of user studies, we also discuss the individual modalities and their influence in interactive applications.

[1]  Wolfgang Maass,et al.  From visual perception to multimodal communication: Incremental route descriptions , 1994 .

[2]  Laurence Nigay,et al.  A Framework for the Combination and Characterization of Output Modalities , 2000, DSV-IS.

[3]  Marilyn A. Walker,et al.  MATCH: An Architecture for Multimodal Dialogue Systems , 2002, ACL.

[4]  J. Sweller,et al.  Reducing cognitive load by mixing auditory and visual presentation modes , 1995 .

[5]  Nicole Yankelovich,et al.  How do users know what to say? , 1996, INTR.

[6]  Wolfgang Wahlster,et al.  Smartkom: multimodal communication with a life- like character , 2001, INTERSPEECH.

[7]  Kristiina Jokinen,et al.  User expectations and real experience on a multimodal interactive system , 2006, INTERSPEECH.

[8]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[9]  Morena Danieli,et al.  Metrics for Evaluating Dialogue Strategies in a Spoken Language System , 1996, ArXiv.

[10]  Jon Oberlander,et al.  A cognitive theory of graphical and linguistic reasoning: logic and implementation. Cognitive Science , 1995 .

[11]  Paul McKevitt.  Integration of Natural Language and Vision Processing: Computational Models and Systems , 1995 .

[12]  Eija Kaasinen,et al.  User needs for location-aware mobile services , 2003, Personal and Ubiquitous Computing.

[13]  Michael Johnston,et al.  Unification-based Multimodal Parsing , 1998, ACL.

[14]  Christian A. Müller,et al.  Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study , 2001, User Modeling.

[15]  Kristiina Jokinen,et al.  Modality fusion in a route navigation system , 2006 .

[16]  Philip R. Cohen,et al.  Intentions in Communication. , 1992 .

[17]  H. Grice.  Logic and conversation , 1975 .

[18]  Markku Turunen,et al.  Jaspis^2 - an architecture for supporting distributed spoken dialogues , 2003, INTERSPEECH.

[19]  Kristiina Jokinen,et al.  On Multimodal Route Navigation in PDAs , 2009 .

[20]  Boris Brandherm,et al.  Adapting Spoken and Visual Output for a Pedestrian Navigation System, based on given Situational Statements , 2003 .

[21]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[22]  Stuart C. Shapiro,et al.  Intelligent multi-media interface technology , 1991 .

[23]  Herbert H. Clark,et al.  Contributing to Discourse , 1989, Cogn. Sci..

[24]  Kristiina Jokinen.  Communicative competence and adaptation in a spoken dialogue system , 2004, INTERSPEECH.

[25]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[26]  Stevan Harnad,et al.  Symbol grounding problem , 1990, Scholarpedia.

[27]  Christian Kray,et al.  Presenting route instructions on mobile devices , 2003, IUI '03.

[28]  W. Maab,et al.  Vitra guide: multimodal route descriptions for computer assisted vehicle navigation , 1993 .

[29]  Dafydd Gibbon,et al.  Handbook of Multimodal and Spoken Dialogue Systems , 2000 .

[30]  Alexander H. Waibel,et al.  Visual tracking for multimodal human computer interaction , 1998, CHI.

[31]  S. Platek,et al.  Common Ground for Spatial Cognition? A Behavioral and fMRI Study of Sex Differences in Mental Rotation and Spatial Working Memory , 2005 .

[32]  Adam Cheyer,et al.  Multimodal Maps: An Agent-Based Approach , 1995, Multimodal Human-Computer Communication.

[33]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[34]  Martin Tomko,et al.  Recursive construction of granular route directions , 2006 .

[35]  Sharon L. Oviatt,et al.  When do we interact multimodally?: cognitive load and multimodal communication patterns , 2004, ICMI '04.

[36]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[37]  Harry Bunt,et al.  Cooperative Multimodal Communication , 2001, Lecture Notes in Computer Science.

[38]  Björn Granström,et al.  Multimodality in Language and Speech Systems , 2002 .

[39]  Sharon Oviatt,et al.  Multimodal interactive maps: designing for human performance , 1997 .

[40]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[41]  Roger K. Moore,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000 .

[42]  Alassane Ndiaye,et al.  Verbmobil From a Software Engineering Point of View: System Design and Software Integration , 2000 .

[43]  Christopher Habel,et al.  Incremental Generation of Multimodal Route Instructions , 2003 .

[44]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[45]  Andrew Lee.  The effect of familiarity in knowledge synchronisation , 2005 .

[46]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[47]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[48]  C. Sidner,et al.  Plans for Discourse , 1988 .

[49]  Robert Dale,et al.  Generating Navigation Information Based on the Driver's Route Knowledge , 2004, COLING 2004.

[50]  Eric Horvitz,et al.  ZoneZoom: map navigation for smartphones with recursive view segmentation , 2004, AVI.

[51]  Markku Turunen,et al.  Adaptive Dialogue Systems - Interaction with Interact , 2002, SIGDIAL Workshop.

[52]  Sebastian Möller.  A new Taxonomy for the Quality of Telephone Services Based on Spoken Dialogue Systems , 2002, SIGDIAL Workshop.

[53]  Anthony Jameson,et al.  Interpreting symptoms of cognitive load in speech input , 1999 .

[54]  Wolfgang Wahlster,et al.  Verbmobil: Foundations of Speech-to-Speech Translation , 2000, Artificial Intelligence.

[55]  Philip R. Cohen,et al.  Unification-based multimodal integration , 1997 .

[56]  E. Zee,et al.  Representing Direction in Language and Space , 2002 .

[57]  Carlo Strapparava,et al.  Modelling and Adapting to Context , 2001, Personal and Ubiquitous Computing.

[58]  Sabine Geldof,et al.  Using Natural Language Generation in Automatic Route Description , 2005, J. Res. Pract. Inf. Technol..

[59]  Ron Burns,et al.  Development of the HRL Route Navigation Dialogue System , 2001, HLT.

[60]  Barbara Tversky,et al.  Places: Points, Planes, Paths, and Portions , 2003 .

[61]  Marilyn A. Walker,et al.  Learning to Predict Problematic Situations in a Spoken Dialogue System: Experiments with How May I Help You? , 2000, ANLP.

[62]  Graeme Hirst,et al.  Collaborating on Referring Expressions , 1991, CL.

[63]  Victor Zue,et al.  GALAXY-II: a reference architecture for conversational system development , 1998, ICSLP.

[64]  H. H. Clark,et al.  Referring as a collaborative process , 1986, Cognition.

[65]  ALISTAIR SUTCLIFFE,et al.  On the effective use and reuse of HCI knowledge , 2000, TCHI.

[66]  Mark T. Maybury,et al.  Intelligent multimedia interfaces , 1994, CHI Conference Companion.