AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

[1]  Jean Vanderdonckt,et al.  Applying model-based techniques to the development of UIs for mobile computers , 2001, IUI '01.

[2]  Anind K. Dey,et al.  XDBrowser: User-Defined Cross-Device Web Page Designs , 2016, CHI.

[3]  Daniel J. Wigdor,et al.  A field study of multi-device workflows in distributed workspaces , 2013, UbiComp.

[4]  Yvonne Rogers,et al.  HuddleLamp: Spatially-Aware Mobile Displays for Ad-hoc Around-the-Table Collaboration , 2014, ITS '14.

[5]  Wendy E. Mackay,et al.  Webstrates: Shareable Dynamic Media , 2015, UIST.

[6]  Mahmoud Naghshineh,et al.  WebSplitter: a unified XML framework for multi-device collaborative Web browsing , 2000, CSCW '00.

[7]  Michael Nebeling,et al.  XDBrowser 2.0: Semi-Automatic Generation of Cross-Device Interfaces , 2017, CHI.

[8]  Moira C. Norrie,et al.  Orchestrating multi-device presentations with OmniPresent , 2017, PerDis.

[9]  Matt Jones,et al.  Mobile collocated interactions: taking an offline break together , 2013, INTR.

[10]  Fabio Paternò,et al.  On-demand cross-device interface components migration , 2010, Mobile HCI.

[11]  Jens Grubert,et al.  Challenges in mobile multi-device ecosystems , 2016, mUX: The Journal of Mobile User Experience.

[12]  Moira C. Norrie,et al.  Interactive development of cross-device user interfaces , 2014, CHI.

[13]  Jeffrey Nichols,et al.  Improving automatic interface generation with smart templates , 2004, IUI '04.

[14]  Brad A. Myers,et al.  Collaboration using multiple PDAs connected to a PC , 1998, CSCW '98.

[15]  Ricardo Langner,et al.  VisTiles: Coordinating and Combining Co-located Mobile Devices for Visual Data Exploration , 2018, IEEE Transactions on Visualization and Computer Graphics.

[16]  Fabio Paternò,et al.  A logical framework for multi-device user interfaces , 2012, EICS '12.

[17]  Antti Oulasvirta,et al.  User Interface Design with Combinatorial Optimization , 2017, Computer.

[18]  Tero Jokela,et al.  A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks , 2015, CHI.

[19]  Harri Oinas-Kukkonen,et al.  Cross-platform service user experience: a field study and an initial framework , 2010, Mobile HCI.

[20]  Martin Porcheron,et al.  Co-curator: designing for mobile ideation in groups , 2016, MindTrek.

[21]  Nicolai Marquardt,et al.  WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications , 2015, CHI.

[22]  Saul Greenberg,et al.  Cross-device interaction via micro-mobility and f-formations , 2012, UIST.

[23]  Roman Rädle,et al.  Codestrates: Literate Computing with Webstrates , 2017, UIST.

[24]  Jeffrey Nichols,et al.  Understanding the Challenges of Designing and Developing Multi-Device Experiences , 2016, Conference on Designing Interactive Systems.

[25]  Niklas Elmqvist,et al.  PolyChrome: A Cross-Device Framework for Collaborative Web Visualization , 2014, ITS '14.

[26]  Jun Rekimoto,et al.  A multiple device approach for supporting whiteboard-based interactions , 1998, CHI.

[27]  Yvonne Rogers,et al.  Spatially-aware or Spatially-agnostic?: Elicitation and Evaluation of User-Defined Cross-Device Interactions , 2015, CHI.

[28]  Roman Rädle,et al.  Connichiwa: A Framework for Cross-Device Web Applications , 2015, CHI Extended Abstracts.

[29]  Antti Oulasvirta,et al.  Improvements to keyboard optimization with integer programming , 2014, UIST.

[30]  Jeffrey Nichols,et al.  Generating remote control interfaces for complex appliances , 2002, UIST '02.

[31]  Andrés Lucero,et al.  Pass-them-around: collaborative use of mobile phones for photo sharing , 2011, CHI.

[32]  Jan Meskens,et al.  Gummy for multi-platform user interface designs: shape me, multiply me, fix me, use me , 2008, AVI '08.

[33]  Andreas Butz,et al.  Organizational Design , 2014, Encyclopedia of Social Network Analysis and Mining.

[34]  Krzysztof Z. Gajos,et al.  SUPPLE: automatically generating user interfaces , 2004, IUI '04.

[35]  Renato Pajarola,et al.  Equalizer: A Scalable Parallel Rendering Framework , 2008, IEEE Transactions on Visualization and Computer Graphics.

[36]  Jean Vanderdonckt,et al.  A toolkit for peer-to-peer distributed user interfaces: concepts, implementation, and applications , 2009, EICS '09.

[37]  Daniel J. Wigdor,et al.  Conductor: enabling and understanding cross-device interaction , 2014, CHI.

[38]  Dieter Schmalstieg,et al.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body , 2015, CHI.

[39]  Michael Nebeling Cross-Device Interfaces: Existing Research, Current Tools, Outlook , 2016, ISS.

[40]  Niklas Elmqvist,et al.  Distributed User Interfaces: State of the Art , 2011, Distributed User Interfaces.

[41]  Jeffrey Nichols,et al.  The Moving Context Kit: Designing for Context Shifts in Multi-Device Experiences , 2017, Conference on Designing Interactive Systems.

[42]  Anirudha Joshi,et al.  Better together: disaggregating mobile services for emergent users , 2017, MobileHCI.

[43]  Scott R. Klemmer,et al.  Automatically adapting web pages to heterogeneous devices , 2011, CHI Extended Abstracts.

[44]  Fabio Paternò,et al.  User interface distribution in multi-device and multi-user environments with dynamically migrating engines , 2014, EICS '14.

[45]  Jorge Gonçalves,et al.  Tandem Browsing Toolkit: Distributed Multi-Display Interfaces with Web Technologies , 2014, PerDis.

[46]  Yang Li,et al.  Weave: Scripting Cross-Device Wearable Interaction , 2015, CHI.

[47]  Kris Luyten,et al.  Distributed user interface elements to support smart interaction spaces , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[48]  Andrew Wilson,et al.  Phone as a pixel: enabling ad-hoc, large-scale displays using mobile devices , 2012, CHI.

[49]  Daniel J. Wigdor,et al.  Panelrama: enabling easy specification of cross-device web applications , 2014, CHI.

[50]  Alan J. Dix,et al.  A taxonomy for and analysis of multi-person-display ecosystems , 2009, Personal and Ubiquitous Computing.