AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures

AirConstellations supports a unique semi-fixed style of cross-device interactions via multiple self-spatially-aware armatures to which users can easily attach (or detach) tablets and other devices. In particular, AirConstellations affords highly flexible and dynamic device formations where the users can bring multiple devices together in-air — with 2–5 armatures poseable in 7DoF within the same workspace — to suit the demands of their current task, social situation, app scenario, or mobility needs. This affords an interaction metaphor where relative orientation, proximity, attaching (or detaching) devices, and continuous movement into and out of ad-hoc ensembles can drive context-sensitive interactions. Yet all devices remain self-stable in useful configurations even when released in mid-air. We explore flexible physical arrangement, feedforward of transition options, and layering of devices in-air across a variety of multi-device app scenarios. These include video conferencing with flexible arrangement of the person-space of multiple remote participants around a shared task-space, layered and tiled device formations with overview+detail and shared-to-personal transitions, and flexible composition of UI panels and tool palettes across devices for productivity applications. A preliminary interview study highlights user reactions to AirConstellations, such as for minimally disruptive device formations, easier physical transitions, and balancing ”seeing and being seen” in remote work.

[1]  Morten Fjeld,et al.  Thaddeus: a dual device interaction space for exploring information visualisation , 2014, NordiCHI.

[2]  Gilles Bailly,et al.  LivingDesktop: Augmenting Desktop Workstation with Actuated Devices , 2016, CHI.

[3]  Jun Rekimoto,et al.  Pick-and-drop: a direct manipulation technique for multiple computer environments , 1997, UIST '97.

[4]  Daniel J. Wigdor,et al.  Conductor: enabling and understanding cross-device interaction , 2014, CHI.

[5]  Patrick Baudisch,et al.  Stitching: pen gestures that span multiple displays , 2004, AVI.

[6]  Tent Mode Interactions: Exploring Collocated Multi-User Interaction on a Foldable Device , 2020, MobileHCI.

[7]  Wendy E. Mackay,et al.  Webstrates: Shareable Dynamic Media , 2015, UIST.

[8]  Kenton O'Hara,et al.  Sensitizing Concepts for Socio-spatial Literacy in HCI , 2017, CHI.

[9]  Sebastian Boring,et al.  Gradual engagement: facilitating information exchange between digital devices as a function of proximity , 2012, ITS.

[10]  Jürgen Steimle,et al.  FoldMe: interacting with double-sided foldable displays , 2012, Tangible and Embedded Interaction.

[11]  Elise van den Hoven,et al.  Crossing the bridge over Norman's Gulf of Execution: revealing feedforward's true identity , 2013, CHI.

[12]  Pattie Maes,et al.  Siftables: towards sensor network user interfaces , 2007, TEI.

[13]  Ehud Sharlin,et al.  The 3D Tractus: a three-dimensional drawing board , 2006, First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06).

[14]  David Kirsh,et al.  The Intelligent Use of Space , 1995, Artif. Intell..

[15]  A. Kendon.  Conducting Interaction: Patterns of Behavior in Focused Encounters , 1990 .

[16]  Saul Greenberg,et al.  Evaluation Strategies for HCI Toolkit Research , 2018, CHI.

[17]  Michael Ortega-Binderberger,et al.  BEXHI: A Mechanical Structure for Prototyping Bendable and EXpandable Handheld Interfaces , 2019, ISS.

[18]  Andrés Lucero,et al.  Pass-them-around: collaborative use of mobile phones for photo sharing , 2011, CHI.

[19]  Ming Li,et al.  Dynamic tiling display: building an interactive display surface using multiple mobile devices , 2012, MUM.

[20]  Teddy Seyed,et al.  SkyHunter: a multi-surface environment for supporting oil and gas exploration , 2013, ITS.

[21]  Daniel J. Wigdor,et al.  Panelrama: enabling easy specification of cross-device web applications , 2014, CHI.

[22]  Sriram Subramanian,et al.  Tilt displays: designing display surfaces with multi-axis tilting and actuation , 2012, Mobile HCI.

[23]  Takuro Yonezawa,et al.  u-Texture: Self-Organizable Universal Panels for Creating Smart Surroundings , 2005, UbiComp.

[24]  Hiroshi Ishii,et al.  ClearBoard: a seamless medium for shared drawing and conversation with eye contact , 1992, CHI.

[25]  Yvonne Rogers,et al.  HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration , 2014, ITS '14.

[26]  Hiroshi Ishii,et al.  The metaDESK: models and prototypes for tangible user interfaces , 1997, UIST '97.

[27]  Jan O. Borchers,et al.  Dynamic portals: a lightweight metaphor for fast object transfer on interactive surfaces , 2011, ITS '11.

[28]  Kris Luyten,et al.  The design of slow-motion feedback , 2014, Conference on Designing Interactive Systems.

[29]  Saul Greenberg,et al.  Cross-device interaction via micro-mobility and f-formations , 2012, UIST.

[30]  Saul Greenberg,et al.  Using digital but physical surrogates to mediate awareness, communication and privacy in media spaces , 1999, Personal Technologies.

[31]  S. Follmer,et al.  Foxels , 2020, Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction.

[32]  Ricardo Langner,et al.  VisTiles: Coordinating and Combining Co-located Mobile Devices for Visual Data Exploration , 2018, IEEE Transactions on Visualization and Computer Graphics.

[33]  Bokyung Lee.  Socio-Spatial Comfort: Using Vision-based Analysis to Inform User-Centred Human-Building Interactions , 2020 .

[34]  Morten Fjeld,et al.  RAMPARTS: Supporting Sensemaking with Spatially-Aware Mobile Interactions , 2016, CHI.

[35]  Michael Nebeling,et al.  XDBrowser 2.0: Semi-Automatic Generation of Cross-Device Interfaces , 2017, CHI.

[36]  William Buxton,et al.  Sentient Data Access via a Diverse Society of Devices , 2003, ACM Queue.

[37]  Harald Reiterer,et al.  Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces , 2014, Personal and Ubiquitous Computing.

[38]  Jakob E. Bardram,et al.  ReticularSpaces: activity-based computing support for physically distributed and collaborative smart spaces , 2012, CHI.

[39]  Sebastian Boring,et al.  Hover Pad: interacting with autonomous and self-actuated displays in space , 2014, UIST.

[40]  Roman Rädle,et al.  PolarTrack: Optical Outside-In Device Tracking that Exploits Display Polarization , 2018, CHI.

[41]  Kent Lyons,et al.  Corona: Positioning Adjacent Device with Asymmetric Bluetooth Low Energy RSSI Distributions , 2015, UIST.

[42]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[43]  Astral , 2019, Proceedings of the 2019 on Designing Interactive Systems Conference.

[44]  Peter Brandl,et al.  Flux: a tilting multi-touch and pen based surface , 2009, CHI Extended Abstracts.

[45]  Nicolai Marquardt,et al.  The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies , 2011, UIST.

[46]  S. Boring,et al.  Tiltstacks , 2018, Proceedings of the 2018 International Conference on Advanced Visual Interfaces.

[47]  Christian Müller-Tomfelde,et al.  Tilted tabletops: In between horizontal and vertical workspaces , 2008, 2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems.

[48]  Paolo Tell,et al.  ActivitySpace: Managing Device Ecologies in an Activity-Centric Configuration Space , 2014, ITS '14.

[49]  A. Roudaut,et al.  Mantis , 2019, Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology.

[50]  Daniel J. Wigdor,et al.  A field study of multi-device workflows in distributed workspaces , 2013, UbiComp.

[51]  Jan O. Borchers,et al.  BendDesk: dragging across the curve , 2010, ITS '10.

[52]  Moira C. Norrie,et al.  XDSession: integrated development and testing of cross-device applications , 2015, EICS.

[53]  Mike Sinclair,et al.  TouchMover: actuated 3D touchscreen with haptic feedback , 2013, ITS.

[54]  Christian Holz,et al.  SurfaceConstellations: A Modular Hardware Platform for Ad-Hoc Reconfigurable Cross-Device Workspaces , 2018, CHI.

[55]  Maxine D. Brown,et al.  The ImmersaDesk and Infinity Wall projection-based virtual reality displays , 1997, COMG.

[56]  Andreas Butz,et al.  Curve: revisiting the digital desk , 2010, NordiCHI.

[57]  Alessio Malizia,et al.  Light on horizontal interactive surfaces: Input space for tabletop computing , 2014, CSUR.

[58]  Nicholas Chen,et al.  Graduate student use of a multi-slate reading system , 2013, CHI.

[59]  Yang Li,et al.  Weave: Scripting Cross-Device Wearable Interaction , 2015, CHI.

[60]  Nicolai Marquardt,et al.  GazeConduits: Calibration-Free Cross-Device Collaboration through Gaze and Touch , 2020, CHI.

[61]  Yvonne Rogers,et al.  Dynamo: a public interactive surface supporting the cooperative sharing and exchange of media , 2003, UIST '03.

[62]  Chris North,et al.  VisPorter: facilitating information sharing for collaborative sensemaking on multiple displays , 2014, Personal and Ubiquitous Computing.

[63]  Nicolas Roussel,et al.  1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems , 2012, CHI.

[64]  William Buxton,et al.  Telepresence: Integrating Shared Task and Person Spaces , 1992 .

[65]  Ian Taylor,et al.  Designing novel interactional workspaces to support face to face consultations , 2003, CHI '03.

[66]  Roel Vertegaal,et al.  DisplayStacks: interaction techniques for stacks of flexible thin-film displays , 2012, CHI.

[67]  Jens Emil Grønbæk,et al.  Proxemic Transitions: Designing Shape-Changing Furniture for Informal Meetings , 2017, CHI.

[68]  William Buxton,et al.  Tilt-Responsive Techniques for Digital Drawing Boards , 2020, UIST.

[69]  Yvonne Rogers,et al.  Using F-formations to analyse spatial patterns of interaction in physical environments , 2011, CSCW.

[70]  John L. Arnott,et al.  Using spatial cues to improve videoconferencing , 1992, CHI.

[71]  Jens Emil Grønbæk,et al.  KirigamiTable: Designing for Proxemic Transitions with a Shape-Changing Tabletop , 2020, CHI.

[72]  Nicholas Chen,et al.  Designing a multi-slate reading environment to support active reading activities , 2012, TCHI.

[73]  Jens Grubert,et al.  HeadPhones: Ad Hoc Mobile Multi-Display Environments through Head Tracking , 2017, CHI.

[74]  Clifton Forlines,et al.  MultiSpace: enabling electronic document micro-mobility in table-centric, multi-device environments , 2006, First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06).

[75]  Hao Jiang,et al.  WeSpace: the design development and deployment of a walk-up and share multi-surface visual collaboration system , 2009, CHI.

[76]  Saul Greenberg,et al.  Proxemic interaction: designing for a proximity and orientation-aware environment , 2010, ITS '10.

[77]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[78]  Ken Hinckley,et al.  Synchronous gestures for multiple persons and computers , 2003, UIST '03.

[79]  John C. Tang,et al.  Three's company: understanding communication channels in three-way distributed collaboration , 2010, CSCW '10.

[80]  Norbert A. Streitz,et al.  Connectables: dynamic coupling of displays for the flexible creation of shared workspaces , 2001, UIST '01.

[81]  William Buxton,et al.  Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display , 2002, UIST '02.

[82]  Roman Rädle,et al.  Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices , 2019, CHI.

[83]  W. Buxton,et al.  Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display , 2002, UIST '02.

[84]  Jun Rekimoto,et al.  Proximal Interactions: A Direct Manipulation Technique for Wireless Networking , 2003, INTERACT.

[85]  Brian P. Bailey,et al.  ARIS: An Interface for Application Relocation in an Interactive Space , 2004, Graphics Interface.

[86]  Jonathan Grudin,et al.  Partitioning digital worlds: focal and peripheral awareness in multiple monitor use , 2001, CHI.

[87]  PickCells , 2019, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.

[88]  Nicolai Marquardt,et al.  Proxemic interactions: the new ubicomp? , 2011, INTR.

[89]  Hemant Bhaskar Surale,et al.  SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and Time , 2020, UIST.

[90]  Hyun-Jean Lee,et al.  Tilting table: a movable screen , 2007, Tangible and Embedded Interaction.

[91]  Kasper Hornbæk,et al.  Tracko: Ad-hoc Mobile 3D Tracking Using Bluetooth Low Energy and Inaudible Signals for Cross-Device Interaction , 2015, UIST.

[92]  M. Fjeld,et al.  AdapTable , 2018, Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces.

[93]  Ravin Balakrishnan,et al.  Codex: a dual screen tablet computer , 2009, CHI.

[94]  Matt Jones,et al.  Mobile collocated interactions: taking an offline break together , 2013, INTR.

[95]  Jörg Müller,et al.  Chained displays: configurations of public displays can be used to influence actor-, audience-, and passer-by behavior , 2012, CHI.

[96]  Hiroshi Ishii,et al.  Bricks: laying the foundations for graspable user interfaces , 1995, CHI '95.

[97]  Xiaojun Bi,et al.  Informal information gathering techniques for active reading , 2012, CHI.

[98]  Darren Leigh,et al.  DiamondTouch: a multi-user touch technology , 2001, UIST '01.

[99]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[100]  Norbert A. Streitz,et al.  i-LAND: an interactive landscape for creativity and innovation , 1999, CHI '99.

[101]  Andrew Wilson,et al.  Phone as a pixel: enabling ad-hoc, large-scale displays using mobile devices , 2012, CHI.

[102]  Dongjun Suh,et al.  UbiWall as a reconfigurable ecosystem of smart blocks , 2011, 5th IEEE International Conference on Digital Ecosystems and Technologies (IEEE DEST 2011).

[103]  Gerd Kortuem,et al.  Supporting device discovery and spontaneous interaction with spatial references , 2009, Personal and Ubiquitous Computing.

[104]  Christian Heath,et al.  Mobility in collaboration , 1998, CSCW '98.

[105]  Hideaki Kuzuoka,et al.  Agora: a remote collaboration system that enables mutual monitoring , 1999, CHI Extended Abstracts.