SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and Time

Knowledge work increasingly spans multiple computing surfaces. Yet in status quo user experiences, content as well as tools, behaviors, and workflows are largely bound to the current device-running the current application, for the current user, and at the current moment in time. SurfaceFleet is a system and toolkit that uses resilient distributed programming techniques to explore cross-device interactions that are unbounded in these four dimensions of device, application, user, and time. As a reference implementation, we describe an interface built using SurfaceFleet that employs lightweight, semi-transparent UI elements known as Applets. Applets appear always-on-top of the operating system, application windows, and (conceptually) above the device itself. But all connections and synchronized data are virtualized and made resilient through the cloud. For example, a sharing Applet known as a Portfolio allows a user to drag and drop unbound Interaction Promises into a document. Such promises can then be fulfilled with content asynchronously, at a later time (or multiple times), from another device, and by the same or a different user.

[1]  Allison Druin,et al.  KidPad: collaborative storytelling for children , 2002, CHI Extended Abstracts.

[2]  D. Norman.  The Design of Everyday Things: Revised and Expanded Edition , 2013 .

[3]  William Buxton,et al.  Sentient Data Access via a Diverse Society of Devices , 2003, ACM Queue.

[4]  Thomas W. Malone,et al.  How do people organize their desks?: Implications for the design of office information systems , 1983, TOIS.

[5]  Yuxi Wang,et al.  SoD-Toolkit: A Toolkit for Interactively Prototyping and Developing Multi-Sensor, Multi-Device Environments , 2015, ITS.

[6]  Jonathan Grudin,et al.  Human Computer Interaction: The Year 2000 and Beyond , 1995, HCI.

[7]  Gerd Kortuem,et al.  Supporting device discovery and spontaneous interaction with spatial references , 2009, Personal and Ubiquitous Computing.

[8]  Nicolai Marquardt,et al.  WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications , 2015, CHI.

[9]  William Buxton,et al.  A marking based interface for collaborative writing , 1993, UIST '93.

[10]  Tony DeRose,et al.  Toolglass and magic lenses: the see-through interface , 1993, SIGGRAPH.

[11]  Moira C. Norrie,et al.  XDKinect: development framework for cross-device interaction using kinect , 2014, EICS.

[12]  Robert Johansen,et al.  Groupware: Computer Support for Business Teams , 1988 .

[13]  Ravin Balakrishnan,et al.  Keepin' it real: pushing the desktop metaphor with physics, piles and the pen , 2006, CHI.

[14]  Jason Nieh,et al.  Heterogeneous Multi-Mobile Computing , 2019, MobiSys.

[15]  Brian P. Bailey,et al.  TEAM STORM: demonstrating an interaction model for working with multiple ideas during creative group work , 2007, C&C '07.

[16]  Catherine C. Marshall,et al.  Saving and using encountered information: implications for electronic periodicals , 2005, CHI.

[17]  Daniel J. Wigdor,et al.  Panelrama: enabling easy specification of cross-device web applications , 2014, CHI.

[18]  Sebastian Burckhardt,et al.  A.M.B.R.O.S.I.A , 2020, Proc. VLDB Endow..

[19]  Allison Druin,et al.  Technology probes: inspiring design for and with families , 2003, CHI '03.

[20]  Moira C. Norrie,et al.  jQMultiTouch: lightweight toolkit and development framework for multi-touch/multi-device web interfaces , 2012, EICS '12.

[21]  George G. Robertson,et al.  Fix and float: object movement by egocentric navigation , 1997, UIST '97.

[22]  William Buxton,et al.  Chunking and Phrasing and the Design of Human-Computer Dialogues (Invited Paper) , 1995, IFIP Congress.

[23]  Paolo Tell,et al.  Activity-centric computing systems , 2019, Commun. ACM.

[24]  Jakob E. Bardram,et al.  ReticularSpaces: activity-based computing support for physically distributed and collaborative smart spaces , 2012, CHI.

[25]  Christine Reid,et al.  The Myth of the Paperless Office , 2003, J. Documentation.

[26]  Dan R. Olsen,et al.  Evaluating user interface systems research , 2007, UIST.

[27]  Randall B. Smith.  Experiences with the Alternate Reality Kit: An Example of the Tension between Literalism and Magic , 1987, IEEE Computer Graphics and Applications.

[28]  William Buxton,et al.  Tracking menus , 2003, UIST '03.

[29]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[30]  Jeff A. Johnson,et al.  The Xerox Star: a retrospective , 1989, Computer.

[31]  Alison Kidd,et al.  The marks are on the knowledge worker , 1994, CHI '94.

[32]  David Kirsh,et al.  The Intelligent Use of Space , 1995, Artif. Intell..

[33]  Olha Bondarenko,et al.  Documents at Hand: Learning from Paper to Improve Digital Technologies , 2005, CHI.

[34]  Saul Greenberg,et al.  Cross-device interaction via micro-mobility and f-formations , 2012, UIST.

[35]  Robert St. Amant,et al.  Characterizing tool use in an interactive drawing environment , 2002, SMARTGRAPH '02.

[36]  Clemens Nylandsted Klokmose,et al.  VIGO: instrumental interaction in multi-surface environments , 2009, CHI.

[37]  Axel Kramer,et al.  Translucent patches—dissolving windows , 1994, UIST '94.

[38]  Daniel J. Wigdor,et al.  A field study of multi-device workflows in distributed workspaces , 2013, UbiComp.

[39]  William Buxton,et al.  WritLarge: Ink Unleashed by Unified Scope, Action, & Zoom , 2017, CHI.

[40]  Michael Boyle,et al.  Customizable physical interfaces for interacting with conventional applications , 2002, UIST '02.

[41]  Carl Gutwin,et al.  A Descriptive Framework of Workspace Awareness for Real-Time Groupware , 2002, Computer Supported Cooperative Work (CSCW).

[42]  Sebastian Boring,et al.  Gradual engagement: facilitating information exchange between digital devices as a function of proximity , 2012, ITS.

[43]  Michel Beaudouin-Lafon,et al.  Towards Unified Principles of Interaction , 2017, CHItaly.

[44]  Jens Emil Grønbæk,et al.  Proxemics Beyond Proximity: Designing for Flexible Social Interaction Through Cross-Device Interaction , 2020, CHI.

[45]  Austin Henderson,et al.  Rooms: the use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface , 1986, TOGS.

[46]  Michael Boyle,et al.  Understanding and Mitigating Display and Presence Disparity in Mixed Presence Groupware , 2005, J. Res. Pract. Inf. Technol..

[47]  Xiaojun Bi,et al.  Informal information gathering techniques for active reading , 2012, CHI.

[48]  Elizabeth D. Mynatt,et al.  It feels better than filing: everyday work experiences in an activity-based computing system , 2009, CHI.

[49]  Jonathan Grudin,et al.  The case against user interface consistency , 1989, CACM.

[50]  Paolo Tell,et al.  ActivitySpace: Managing Device Ecologies in an Activity-Centric Configuration Space , 2014, ITS '14.

[51]  Daniel G. Bobrow,et al.  WYSIWIS revised: early experiences with multiuser interfaces , 1987, TOIS.

[52]  Allison Druin,et al.  Single display groupware: a model for co-present collaboration , 1999, CHI '99.

[53]  Jagoda Walny.  Thinking with Sketches: Leveraging Everyday Use of Visuals for Information Visualization , 2016 .

[54]  Paul Dourish,et al.  Re-place-ing space: the roles of place and space in collaborative systems , 1996, CSCW '96.

[55]  Wendy E. Mackay,et al.  Reification, polymorphism and reuse: three principles for designing visual interfaces , 2000, AVI '00.

[56]  Jun Rekimoto,et al.  Time-machine computing: a time-centric approach for the information environment , 1999, UIST '99.

[57]  Mirjam Augstein,et al.  Domino , 2018, Proc. ACM Hum. Comput. Interact..

[58]  Nicolai Marquardt,et al.  Proxemic interactions: the new ubicomp? , 2011, INTR.

[59]  Daniel J. Wigdor,et al.  Conductor: enabling and understanding cross-device interaction , 2014, CHI.

[60]  Badrish Chandramouli,et al.  CRA: Enabling Data-Intensive Applications in Containerized Environments , 2019, 2019 IEEE 35th International Conference on Data Engineering (ICDE).

[61]  W. Keith Edwards,et al.  LiquidText: a flexible, multitouch environment to support active reading , 2011, CHI.

[62]  Robert St. Amant,et al.  HabilisDraw DT: a bimanual tool-based direct manipulation drawing environment , 2004, CHI EA '04.

[63]  Saul Greenberg,et al.  Evaluation Strategies for HCI Toolkit Research , 2018, CHI.

[64]  Nicolai Marquardt,et al.  The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies , 2011, UIST.

[65]  Yang Li,et al.  Weave: Scripting Cross-Device Wearable Interaction , 2015, CHI.

[66]  Donald A. Schön.  Designing as reflective conversation with the materials of a design situation , 1992, Knowl. Based Syst..

[67]  Daniel G. Bobrow,et al.  WYSIWIS revised: early experiences with multi-user interfaces , 1986, CSCW.

[68]  James D. Hollan,et al.  Local tools: an alternative to tool palettes , 1996, UIST '96.

[69]  Michel Beaudouin-Lafon,et al.  Instrumental interaction: an interaction model for designing post-WIMP user interfaces , 2000, CHI.

[70]  Jonathan Grudin,et al.  Partitioning digital worlds: focal and peripheral awareness in multiple monitor use , 2001, CHI.

[71]  Christian Heath,et al.  Mobility in collaboration , 1998, CSCW '98.

[72]  Saul Greenberg,et al.  GROUPKIT: a groupware toolkit for building real-time conferencing applications , 1992, CSCW '92.

[73]  Richard Bentley,et al.  Supporting Collaborative Information Sharing with the World Wide Web: The BSCW Shared Workspace System , 1995, WWW Spring 1995.

[74]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[75]  M. Angela Sasse,et al.  "Stuff goes into the computer and doesn't come out": a cross-tool study of personal information management , 2004, CHI.

[76]  Paul Dourish,et al.  Portholes: supporting awareness in a distributed work group , 1992, CHI.

[77]  Wendy E. Mackay,et al.  Webstrates: Shareable Dynamic Media , 2015, UIST.