Challenges in mobile multi-device ecosystems

BackgroundCoordinated multi-display environments from the desktop, second-screen to gigapixel display walls are increasingly common. Personal and intimate mobile and wearable devices such as head-mounted displays, smartwatches, smartphones and tablets are rarely part of such multi-device ecosystems.MethodsWe conducted a literature research and an expert survey to identify challenges in mobile multi-device ecosystems.ResultsWe present grounded challenges relevant for the design, development and use of mobile multi-device environments as well as opportunities for future research. While our surveys indicated that a large number of challenges have been identified, there seems to be little agreement among experts on the importance of individual challenges.ConclusionBy presenting the identified challenges, we contribute to a better understanding about factors that impede the creation and use of mobile multi-device ecosystems and hope to contribute to shaping the research agenda on interacting with those systems.

[1]  Andy Brown,et al.  HCI over multiple screens , 2014, CHI Extended Abstracts.

[2]  Andrés Lucero,et al.  MobiComics: collaborative use of mobile phones and large displays for public expression , 2012, Mobile HCI.

[3]  Jens Grubert,et al.  Natural feature tracking in JavaScript , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[4]  Daniel J. Simons,et al.  Inattentional blindness , 2007, Scholarpedia.

[5]  Kent Lyons,et al.  Corona: Positioning Adjacent Device with Asymmetric Bluetooth Low Energy RSSI Distributions , 2015, UIST.

[6]  Andrés Lucero,et al.  FlexiGroups: binding mobile devices for collaborative interactions in medium-sized groups with device touch , 2014, MobileHCI '14.

[7]  Mark Billinghurst,et al.  Wearable Devices: New Ways to Manage Information , 1999, Computer.

[8]  Eli Pariser,et al.  The Filter Bubble: What the Internet Is Hiding from You , 2011 .

[9]  Ashweeni Kumar Beeharee,et al.  The case of the missed icon: change blindness on mobile devices , 2012, CHI.

[10]  Jennifer Pearson,et al.  It's About Time: Smartwatches as Public Displays , 2015, CHI.

[11]  Hans-Werner Gellersen,et al.  How groups of users associate wireless devices , 2013, CHI.

[12]  Jens Grubert,et al.  Comparative User Study of two See-through Calibration Methods , 2010, VR.

[13]  Per Ola Kristensson,et al.  AwToolkit: attention-aware user interface widgets , 2014, AVI.

[14]  Moira C. Norrie,et al.  Interactive development of cross-device user interfaces , 2014, CHI.

[15]  Sriram Subramanian,et al.  Holographic acoustic elements for manipulation of levitated objects , 2015, Nature Communications.

[16]  A. Kendon.  Conducting Interaction: Patterns of Behavior in Focused Encounters , 1990 .

[17]  Yvonne Rogers,et al.  Enticing People to Interact with Large Public Displays in Public Spaces , 2003, INTERACT.

[18]  A. W. Roscoe,et al.  Two heads are better than one: security and usability of device associations in group scenarios , 2010, SOUPS.

[19]  Hans-Werner Gellersen,et al.  How users associate wireless devices , 2011, CHI.

[20]  W. Charman.  The eye in focus: accommodation and presbyopia , 2008, Clinical & experimental optometry.

[21]  William Mackaness,et al.  Quantifying Urban Visibility Using 3D Space Syntax , 2007 .

[22]  Ellen Yi-Luen Do,et al.  Don't mind me touching my wrist: a case study of interacting with on-body technology in public , 2013, ISWC '13.

[23]  René Mayrhofer,et al.  A Survey of User Interaction for Spontaneous Device Association , 2014, CSUR.

[24]  Johannes Schöning,et al.  Cross-Surface: Workshop on Interacting with Multi-Device Ecologies in the Wild , 2015, ITS.

[25]  Allison B. Sekuler,et al.  Age related differences in learning with the useful field of view , 2006, Vision Research.

[26]  Per Ola Kristensson,et al.  Subtle gaze-dependent techniques for visualising display changes in multi-display environments , 2013, IUI '13.

[27]  Jesper Kjeldskov,et al.  The interaction space of a multi-device, multi-user music experience , 2012, NordiCHI.

[28]  Tad Hirsch,et al.  Exploring Mobile Devices as Personal Public Displays , 2015, MobileHCI.

[29]  Ying-Chao Tung,et al.  User-Defined Game Input for Smart Glasses in Public Space , 2015, CHI.

[30]  Aaron J. Quigley,et al.  The cost of display switching: a comparison of mobile, large display and hybrid UI configurations , 2012, AVI.

[31]  Pourang Irani,et al.  The personal cockpit: a spatial interface for effective task switching on head-worn displays , 2014, CHI.

[32]  Daniel J. Wigdor,et al.  Panelrama: enabling easy specification of cross-device web applications , 2014, CHI.

[33]  Alan J. Dix,et al.  A taxonomy for and analysis of multi-person-display ecosystems , 2009, Personal and Ubiquitous Computing.

[34]  Jock D. Mackinlay,et al.  Proceedings of the 5th annual ACM symposium on User interface software and technology , 1992 .

[35]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[36]  A. Strauss,et al.  Basics of Qualitative Research , 1992 .

[37]  Tero Jokela,et al.  A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks , 2015, CHI.

[38]  Michael E. Holmes,et al.  Visual attention to television programs with a second-screen application , 2012, ETRA '12.

[39]  Aaron Quigley,et al.  From GUI to UUI: Interfaces for Ubiquitous Computing , 2010, Ubicomp 2010.

[40]  P. Wiffen.  Change blindness , 2018, European journal of hospital pharmacy. Science and practice.

[41]  Saul Greenberg,et al.  Proxemic Interactions: From Theory to Practice , 2015, Proxemic Interactions.

[42]  Sebastian Boring,et al.  Gradual engagement: facilitating information exchange between digital devices as a function of proximity , 2012, ITS.

[43]  Elizabeth D. Mynatt,et al.  Semi-public displays for small, co-located groups , 2003, CHI '03.

[44]  Hiroshi Ishii,et al.  Visions and visioning in CHI: CHI 2013 special interest group meeting , 2013, CHI Extended Abstracts.

[45]  Vassilis Kostakos.  The big hole in HCI research , 2015, Interactions.

[46]  Chris Harrison,et al.  Implications of location and touch for on-body projected interfaces , 2014, Conference on Designing Interactive Systems.

[47]  Sriram Subramanian,et al.  Identifying suitable projection parameters and display configurations for mobile true-3D displays , 2014, MobileHCI '14.

[48]  Arun Kumar,et al.  Pairing devices for social interactions: a comparative usability evaluation , 2011, CHI.

[49]  Xing-Dong Yang,et al.  Gluey: Developing a Head-Worn Display Interface to Unify the Interaction Experience in Distributed Display Environments , 2015, MobileHCI.

[50]  Kasper Hornbæk,et al.  Tracko: Ad-hoc Mobile 3D Tracking Using Bluetooth Low Energy and Inaudible Signals for Cross-Device Interaction , 2015, UIST.

[51]  Dieter Schmalstieg,et al.  Experiences with the Impact of Tracking Technology in Mobile Augmented Reality Evaluations , 2012 .

[52]  Andrés Lucero,et al.  Pass-them-around: collaborative use of mobile phones for photo sharing , 2011, CHI.

[53]  Andrés Lucero,et al.  A comparative evaluation of touch-based methods to bind mobile devices for collaborative interactions , 2013, CHI.

[54]  岩井 大輔.  IEEE VR 2010 , 2010 .

[55]  Li-Wei Chan,et al.  CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring , 2015, UIST.

[56]  Oscar Tomico,et al.  Mobile Collocated Interactions With Wearables , 2015, MobileHCI Adjunct.

[57]  Chris Harrison.  Appropriated Interaction Surfaces , 2010, Computer.

[58]  Christian Heath,et al.  Mobility in collaboration , 1998, CSCW '98.

[59]  Pourang Irani,et al.  Ethereal planes: a design framework for 2D information space in 3D mixed reality environments , 2014, SUI.

[60]  Dieter Schmalstieg,et al.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body , 2015, CHI.

[61]  Xiang 'Anthony' Chen,et al.  Air+touch: interweaving touch & in-air gestures , 2014, UIST.

[62]  Michael Evans,et al.  Mediating Attention for Second Screen Companion Content , 2015, CHI.

[63]  Ken Hinckley,et al.  A survey of design issues in spatial input , 1994, UIST '94.

[64]  C. Chabris,et al.  Gorillas in Our Midst: Sustained Inattentional Blindness for Dynamic Events , 1999, Perception.

[65]  Kristof Van Laerhoven,et al.  From Mobile to Wearable: Using Wearable Devices to Enrich Mobile Interaction , 2015, MobileHCI Adjunct.

[66]  Steve Benford,et al.  Designing the spectator experience , 2005, CHI.

[67]  Radu-Daniel Vatavu,et al.  Visual attention measures for multi-screen TV , 2014, TVX '14.

[68]  Andrés Lucero,et al.  Connecting devices for collaborative interactions , 2015, Interactions.

[69]  Matt Jones,et al.  Designing attention for multi-screen TV experiences , 2015, BCS HCI.

[70]  Marc Busch,et al.  Money on the move workload, usability and technology acceptance of second-screen atm-interactions , 2013, MobileHCI '13.

[71]  Aaron J. Quigley,et al.  Factors influencing visual attention switch in multi-display user interfaces: a survey , 2012, PerDis.

[72]  Dieter Schmalstieg,et al.  Playing it real again: a repeated evaluation of magic lens and static peephole interfaces in public space , 2013, MobileHCI '13.

[73]  Sebastian Boring,et al.  Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct , 2015, International Conference on Human-Computer Interaction with Mobile Devices and Services.

[74]  Dominik Schmidt,et al.  A cross-device interaction style for mobiles and surfaces , 2012, DIS '12.

[75]  Matthias Nießner,et al.  RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces , 2014, CHI.

[76]  Nicolai Marquardt,et al.  WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications , 2015, CHI.

[77]  Albrecht Schmidt,et al.  Requirements and design space for interactive public displays , 2010, ACM Multimedia.

[78]  Nicolai Marquardt,et al.  Proxemic interactions: the new ubicomp? , 2011, INTR.

[79]  J. Hodges,et al.  Attention and executive deficits in Alzheimer's disease. A critical review. , 1999, Brain : a journal of neurology.

[80]  Wendy E. Mackay,et al.  Body-centric design space for multi-surface interaction , 2013, CHI.

[81]  Saul Greenberg,et al.  The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface , 2011, INTERACT.

[82]  Sriram Subramanian,et al.  Rendering volumetric haptic shapes in mid-air using ultrasound , 2014, ACM Trans. Graph..

[83]  Eric Horvitz,et al.  Foreground and background interaction with sensor-enhanced mobile devices , 2005, TCHI.

[84]  Daniel J. Wigdor,et al.  A field study of multi-device workflows in distributed workspaces , 2013, UbiComp.

[85]  Yvonne Rogers,et al.  HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration , 2014, ITS '14.

[86]  Da-Yuan Huang,et al.  Cyclops: Wearable and Single-Piece Full-Body Gesture Input Devices , 2015, CHI.

[87]  Chi Tai Dang,et al.  A framework towards challenges and issues of multi-surface environments , 2015 .

[88]  Marcos Serrano.  Smartwatches + Head-worn Displays: the 'new' Smartphone Chi 2015 " Mobile Collocated Interactions " Workshop , 2015 .

[89]  E. Hall,et al.  The Hidden Dimension , 1970 .

[90]  Bill Hillier,et al.  Space is the machine: A configurational theory of architecture , 1996 .

[91]  Yvonne Rogers,et al.  Using F-formations to analyse spatial patterns of interaction in physical environments , 2011, CSCW.

[92]  Tovi Grossman,et al.  Candid Interaction: Revealing Hidden Mobile and Wearable Computing Activities , 2015, UIST.

[93]  Xiang 'Anthony' Chen,et al.  Duet: exploring joint interactions on a smart phone and a smart watch , 2014, CHI.

[94]  Yang Li,et al.  Weave: Scripting Cross-Device Wearable Interaction , 2015, CHI.

[95]  Xing-Dong Yang,et al.  Smartwatches + Head-Worn Displays: the "New" Smartphone , 2015 .

[96]  Yvonne Rogers,et al.  Spatially-aware or Spatially-agnostic?: Elicitation and Evaluation of User-Defined Cross-Device Interactions , 2015, CHI.

[97]  Roman Rädle,et al.  Connichiwa: A Framework for Cross-Device Web Applications , 2015, CHI Extended Abstracts.

[98]  Johannes Schöning,et al.  Visual separation in mobile multi-display environments , 2011, UIST.

[99]  J. Pateman,et al.  What is Social Exclusion , 2016 .

[100]  Daniel J. Wigdor,et al.  Conductor: enabling and understanding cross-device interaction , 2014, CHI.

[101]  Ann Morrison,et al.  Playing it real: magic lens and static peephole interfaces for games in a public space , 2012, Mobile HCI.

[102]  Saul Greenberg,et al.  Cross-device interaction via micro-mobility and f-formations , 2012, UIST.

[103]  Roman Rädle,et al.  Proxemics in Human-Computer Interaction (Dagstuhl Seminar 13452) , 2013, Dagstuhl Reports.