Physical mobile interactions : mobile devices as pervasive mediators for interactions with the real world

So far, mobile devices have mainly been used for interactions between the user, the device and the used service without considering the context of use. However, during the last years we have seen a huge interest in industry and academia in using mobile devices for interactions with things, places and people in the real world, termed physical mobile interactions in this thesis. Until now there has been no comprehensive analysis of these interaction techniques and no user studies have been conducted to analyze when which interaction technique is preferred by which users. Furthermore there is no comprehensive framework available which can be reused by application developers to integrate such interactions into their applications, and no specific methods and best practices have been reported that can be of use when developing physical mobile interactions and applications. This dissertation presents the first comprehensive analysis and classification of physical mobile interactions. Furthermore a mature framework was developed that provides various implementations of four different interaction techniques. These four physical mobile interaction techniques were then used in five different prototypes and analysed in five different user studies. The results concern the advantages and disadvantages of these interaction techniques as seen by potential users. This work also reports experiences, guidelines, methods and best practices that simplify the process of developing physical mobile interactions and applications. Furthermore this dissertation provides an analysis of privacy aspects in mobile interactions with public displays, presents the novel interaction technique rotating compass and the first concept of using the mobile device for direct touch-based interaction with dynamic displays.

[1]  Jock D. Mackinlay,et al.  A morphological analysis of the design space of input devices , 1991, TOIS.

[2]  Guanling Chen,et al.  A Survey of Context-Aware Mobile Computing Research , 2000 .

[3]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.

[4]  Joseph A. Paradiso,et al.  The FindIT Flashlight: Responsive Tagging Based on Optically Triggered Microprocessor Wakeup , 2002, UbiComp.

[5]  L. Paletta,et al.  Mobile Vision for Ambient Learning in Urban Environments , 2004 .

[6]  Emmanuel Dubois,et al.  ASUR++: Supporting the design of mobile mixed systems , 2003, Interact. Comput..

[7]  Kori Inkpen Quinn,et al.  Just point and click?: using handhelds to interact with paper maps , 2005, Mobile HCI.

[8]  Axel Küpper Location-based Services: Fundamentals and Operation , 2005 .

[9]  David K. Vawdrey,et al.  RF Rendez-Blue: reducing power and inquiry costs in Bluetooth-enabled mobile systems , 2002, Proceedings. Eleventh International Conference on Computer Communications and Networks.

[10]  Brad A. Myers Using handhelds for wireless remote control of PCs and appliances , 2005, Interact. Comput..

[11]  Michael Rohs,et al.  USING CAMERA-EQUIPPED MOBILE PHONES FOR INTERACTING WITH REAL-WORLD OBJECTS , 2004 .

[12]  Nuno Jardim Nunes,et al.  Towards a UML profile for interaction design: the Wisdom approach , 2000, UML.

[13]  Albrecht Schmidt,et al.  Multi-Sensor Context-Awareness in Mobile Devices and Smart Artifacts , 2002, Mob. Networks Appl..

[14]  W. W. Royce,et al.  Managing the development of large software systems , 1970 .

[15]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[16]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[17]  Heikki Ailisto,et al.  Realising Physical Selection for Mobile Devices , 2003 .

[18]  Norman W. Paton,et al.  UMLi: The Unified Modeling Language for Interactive Applications , 2000, UML.

[19]  James D. Foley,et al.  The human factors of computer graphics interaction techniques , 1984, IEEE Computer Graphics and Applications.

[20]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[21]  Graziano Bertini,et al.  Remote Control of Home Automation Systems with Mobile Devices , 2002, Mobile HCI.

[22]  Albrecht Schmidt,et al.  Towards a Playful User Interface for Home Entertainment Systems , 2004, EUSAI.

[23]  I. Scott MacKenzie,et al.  Touchpad-based remote control devices , 1998, CHI Conference Summary.

[24]  W. Buxton Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[25]  Dieter Schmalstieg,et al.  Towards Massively Multi-user Augmented Reality on Handheld Devices , 2005, Pervasive.

[26]  Panos Markopoulos,et al.  GentleGuide: An Exploration of Haptic Output for Indoors Pedestrian Guidance , 2003, Mobile HCI.

[27]  I. Scott MacKenzie,et al.  An Isometric Joystick as a Pointing Device for Handheld Information Terminals , 2001, Graphics Interface.

[28]  Jan Borchers,et al.  iStuff Mobile : prototyping interactions for mobile phones in interactive spaces , 2006 .

[29]  Dominik Schmidt,et al.  Using Mobile Phones for Domain Specific Information Appliances , 2005 .

[30]  Jonathan J. Cadiz,et al.  Interaction Issues in Context-Aware Intelligent Environments , 2001, Hum. Comput. Interact..

[31]  J. Boyd Here comes the wallet phone [wireless credit card] , 2005, IEEE Spectrum.

[32]  Albrecht Schmidt,et al.  Privacy and Curiosity in Mobile Interactions with Public Displays. , 2007 .

[33]  Ichiro Satoh Linking physical worlds to logical worlds with mobile agents , 2004, IEEE International Conference on Mobile Data Management, 2004. Proceedings. 2004.

[34]  Clare-Marie Karat,et al.  CHI 98 Conference Summary on Human Factors in Computing Systems , 1998, CHI Conference Summary.

[35]  Albrecht Schmidt,et al.  JaGD – An Photo Oriented Learning and Gaming Platform for Mobile Phones , 2005 .

[36]  Michael Rohs,et al.  Linking physical and virtual worlds with visual markers and handheld devices , 2005 .

[37]  Keith Cheverst,et al.  Developing a context-aware electronic tourist guide: some issues and experiences , 2000, CHI.

[38]  Meredith Ringel Morris,et al.  iStuff: a physical user interface toolkit for ubiquitous computing environments , 2003, CHI '03.

[39]  Johannes Schöning,et al.  Interaction of mobile camera devices with physical maps , 2006 .

[40]  Charles D. Knutson,et al.  Rapid heterogeneous ad hoc connection establishment: accelerating Bluetooth inquiry using IrDA , 2002, 2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No.02TH8609).

[41]  Jan Van den Bergh,et al.  Towards modeling context-sensitive interactive applications: the context-sensitive user interface profile (CUP) , 2005, SoftVis '05.

[42]  Roy Want,et al.  Bridging physical and virtual worlds with electronic tags , 1999, CHI '99.

[43]  H. Hussmann,et al.  Policy Based Adaptive Services for Mobile Commerce , 2005, Second IEEE International Workshop on Mobile Commerce and Services.

[44]  Andrew D. Wilson Multimodal Sensing for Explicit and Implicit Interaction , 2005 .

[45]  Darren Leigh,et al.  RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors , 2004, SIGGRAPH 2004.

[46]  Hiroshi Ishii,et al.  The metaDESK: models and prototypes for tangible user interfaces , 1997, UIST '97.

[47]  Heikki Ailisto,et al.  A user interaction paradigm for physical browsing and near-object control based on tags , 2003 .

[48]  Alexander De Luca,et al.  Automatic form filling on mobile devices , 2008, Pervasive Mob. Comput..

[49]  Michael Boyle,et al.  PDAs and shared public displays: Making personal information public, and public information personal , 1999, Personal Technologies.

[50]  Richard Sharp,et al.  Using visual tags to bypass Bluetooth device discovery , 2005, MOCO.

[51]  Albrecht Schmidt,et al.  The rotating compass: a novel interaction technique for mobile navigation , 2005, CHI EA '05.

[52]  Jeffrey Nichols,et al.  Generating remote control interfaces for complex appliances , 2002, UIST '02.

[53]  Wendy E. Mackay,et al.  Perspectives: trialogue on design (of) , 2001, INTR.

[54]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[55]  Albrecht Schmidt,et al.  An Analysis of the Usage of Mobile Phones for Personalized Interactions with Ubiquitous Public Displays , 2004 .

[56]  Lucia Terrenghi,et al.  The physical user interface profile (PUIP): modelling mobile interactions with the real world , 2005, TAMODIA '05.

[57]  Suguru Higashino,et al.  C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal , 2004, UIST '04.

[58]  Jeffrey Nichols,et al.  Interacting at a distance: measuring the performance of laser pointers and other devices , 2002, CHI.

[59]  Eric Horvitz,et al.  Sensing techniques for mobile interaction , 2000, UIST '00.

[60]  William Buxton,et al.  Lexical and pragmatic considerations of input structures , 1983, COMG.

[61]  Shojiro Nishio,et al.  Rajicon: remote PC GUI operations via constricted mobile interfaces , 2002, MobiCom '02.

[62]  K CardStuart,et al.  A morphological analysis of the design space of input devices , 1991 .

[63]  Scott Weiss Handheld usability , 2002, DIS '02.

[64]  J. Bartlett WRL Research Report 2000 / 3 Rock ’ n ’ Scroll is Here to Stay , 2000 .

[65]  Albrecht Schmidt,et al.  Supporting Mobile Service Usage through Physical Mobile Interaction , 2007, Fifth Annual IEEE International Conference on Pervasive Computing and Communications (PerCom'07).

[66]  J. F. Bartlett,et al.  Rock 'n' Scroll Is Here to Stay , 2000 .

[67]  Albrecht Schmidt,et al.  Physical posters as gateways to context-aware services for mobile devices , 2004, Sixth IEEE Workshop on Mobile Computing Systems and Applications.

[68]  Keith Cheverst,et al.  Exploring bluetooth based mobile phone interaction with the hermes photo display , 2005, Mobile HCI.

[69]  A. Schmidt,et al.  Mobile Service Interaction with the Web of Things , .

[70]  Gregory D. Abowd,et al.  Cyberguide: A mobile context‐aware tour guide , 1997, Wirel. Networks.

[71]  Clayton Lewis,et al.  TASK-CENTERED USER INTERFACE DESIGN A Practical Introduction , 2006 .

[72]  Mark Weiser The computer for the 21st Century , 2002 .

[73]  Paul A. Beardsley,et al.  RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors , 2004, ACM Trans. Graph..

[74]  Bill N. Schilit,et al.  Place Lab: Device Positioning Using Radio Beacons in the Wild , 2005, Pervasive.

[75]  Enrico Rukzio,et al.  Mobile Services for Near Field Communication , 2007 .

[76]  Jadwiga Indulska,et al.  Modeling Context Information in Pervasive Computing Systems , 2002, Pervasive.

[77]  David Abramson,et al.  Are you interested in computers and electronics? , 2000, ACSE '00.

[78]  Mark Weiser The computer for the 21st century , 1991 .

[79]  Elizabeth D. Mynatt,et al.  Semi-public displays for small, co-located groups , 2003, CHI '03.

[80]  Massimo Paolucci,et al.  Supporting Service Interaction in the Real World , 2006 .

[81]  Patrick Olivier,et al.  Immersive video as a rapid prototyping and evaluation tool for mobile and ambient applications , 2006, Mobile HCI.

[82]  Oliver Bimber,et al.  PhoneGuide: museum guidance supported by on-device object recognition on mobile phones , 2005, MUM '05.

[83]  Christian Kray,et al.  Presenting route instructions on mobile devices , 2003, IUI '03.

[84]  Brad A. Myers,et al.  Collaboration using multiple PDAs connected to a PC , 1998, CSCW '98.

[85]  Turkka Keinonen,et al.  Mobile Usability: How Nokia Changed the Face of the Mobile Phone , 2003 .

[86]  Vic Callaghan,et al.  Mobile Interaction in Smart Environments , 2006 .

[87]  Per Persson,et al.  Nokia sensor: from research to product , 2005, DUX '05.

[88]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[89]  Albrecht Schmidt,et al.  An Experimental Comparison of Physical Mobile Interaction Techniques: Touching, Pointing and Scanning , 2006, UbiComp.

[90]  Fabio Paternò,et al.  ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models , 1997, INTERACT.

[91]  Koji Tsukada,et al.  ActiveBelt: Belt-Type Wearable Tactile Display for Directional Navigation , 2004, UbiComp.

[92]  Albrecht Schmidt,et al.  Mobile Interaction with the Internet of Things , 2006 .

[93]  Kevin B. Wright,et al.  Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services , 2006, J. Comput. Mediat. Commun..

[94]  Liviu Iftode,et al.  Smart Phone: an embedded system for universal interactions , 2004, Proceedings. 10th IEEE International Workshop on Future Trends of Distributed Computing Systems, 2004. FTDCS 2004..

[95]  Wolfgang Wahlster,et al.  A resource-adaptive mobile navigation system , 2002, IUI '02.

[96]  Michael Rohs,et al.  The smart phone: a ubiquitous input device , 2006, IEEE Pervasive Computing.

[97]  Bill Serra,et al.  People, Places, Things: Web Presence for the Real World , 2002, Mob. Networks Appl..

[98]  Adrian David Cheok,et al.  22nd International Conference on Human-Computer Interaction with Mobile Devices and Services , 2007, Lecture Notes in Computer Science.

[99]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[100]  Kori Inkpen Quinn,et al.  Marked-up maps: combining paper maps and electronic information resources , 2006, Personal and Ubiquitous Computing.

[101]  E. Rukzio,et al.  A Framework for Mobile Interactions with the Physical World , 2005 .

[102]  Michael Beigl,et al.  The particle computer system , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..