Artefact: A UML-based framework for model-driven development of interactive surface prototypes

While interactive surface prototypes may be highly application-specific, existing prototypes hint at common, recurring design considerations. Given the rapid accumulation of near-identical prototypes, there is a need to promote design reuse. In this context, existing research prototypes motivate abstracting generic structures, architectural views, and descriptions to inform future designs. This paper proposes Artefact: a UML-based framework for model-driven development of interactive surface prototypes. We define flexible base models using existing research prototypes: initial hardware and middleware abstractions to support developers in the early design stages. For validation, we use the proposed framework to capture existing research prototypes. We then conduct an interview study to learn expert perceptions towards the captured model representations. Our initial findings highlight three significant benefits: (1) an accessible graphical syntax with unambiguous model representation, (2) a system for capturing arbitrary technical specifications, and (3) flexible model representation with consistent notation. While we can not draw any absolute conclusions, initial results suggest benefits in the model-driven approach.

[1]  Martin Becker,et al.  Variability realization in UML/SysML models , 2020, SPLC.

[2]  Martin Kaltenbrunner,et al.  The TUIO 2.0 Protocol , 2018, Proc. ACM Hum. Comput. Interact..

[3]  Florian Echtler SurfaceStreams: A Content-Agnostic Streaming Toolkit for Interactive Surfaces , 2018, UIST.

[4]  Daniel Vogel,et al.  An Evaluation of Touch Input at the Edge of a Table , 2019, CHI.

[5]  Martin Kaltenbrunner,et al.  Shared Infrastructures for Tangible Tabletops & Interactive Surfaces , 2015, ITS.

[6]  Yoonjung Park,et al.  Adaptive projection augmented reality with object recognition based on deep learning , 2019, IUI Companion.

[7]  Martin Kaltenbrunner An Abstraction Framework for Tangible Interactive Surfaces , 2018 .

[8]  Cagatay Basdogan,et al.  HapTable: An Interactive Tabletop Providing Online Haptic Feedback for Touch Gestures , 2019, IEEE Transactions on Visualization and Computer Graphics.

[9]  Martin Kaltenbrunner reacTIVision and TUIO: a tangible tabletop toolkit , 2009, ITS '09.

[10]  Mario Piattini,et al.  A Systematic Literature Review on the Quality of UML Models , 2009 .

[11]  Gregorio Robles,et al.  Practices and Perceptions of UML Use in Open Source Projects , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[12]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[13]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[14]  Hiroshi Ishii,et al.  The metaDESK: models and prototypes for tangible user interfaces , 1997, UIST '97.

[15]  Shuchang Xu,et al.  Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor , 2019, UIST.

[16]  Enrico Rukzio,et al.  UbiBeam: An Interactive Projector-Camera System for Domestic Deployment , 2014, ITS '14.

[17]  Karthik Ramani,et al.  Extended multitouch: recovering touch posture and differentiating users using a depth camera , 2012, UIST.

[18]  Robert B. France,et al.  Repository for Model Driven Development (ReMoDD) , 2006, 2012 34th International Conference on Software Engineering (ICSE).

[19]  H. James Nelson,et al.  Research Review: A Systematic Literature Review on the Quality of UML Models , 2011, J. Database Manag..

[20]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[21]  Yoichi Sato,et al.  Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system , 2001, TCHI.

[22]  Jiayuan Wang,et al.  MagicPAPER: tabletop interactive projection device based on tangible interaction , 2019, SIGGRAPH Posters.

[23]  Florian Echtler SurfaceStreams , 2018, The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings.

[24]  Raphael Wimmer,et al.  The Interactive Dining Table, or Pass the Weather Widget, Please , 2014, ITS '14.

[25]  Leandro Flores da Silva,et al.  Evaluating usefulness, ease of use and usability of an UML-based Software Product Line Tool , 2020, SBES.

[26]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[27]  Colin Atkinson,et al.  Model-Driven Development: A Metamodeling Foundation , 2003, IEEE Softw..

[28]  Andrew D. Wilson PlayAnywhere: a compact interactive tabletop projection-vision system , 2005, UIST.

[29]  Matthias Bues,et al.  ProDesk: An Interactive Ubiquitous Desktop Surface , 2017, ISS.

[30]  Damiano Torre,et al.  UML Consistency Rules: a Case Study with Open-Source UML Models , 2020, FormaliSE@ICSE.

[31]  Otmar Hilliges,et al.  Steerable augmented reality with the beamatron , 2012, UIST.

[32]  Gregorio Robles,et al.  The quest for open source projects that use UML: mining GitHub , 2016, MoDELS.

[33]  Robert Xiao,et al.  Supporting Responsive Cohabitation Between Virtual Interfaces and Physical Objects on Everyday Surfaces , 2017, PACMHCI.

[34]  Andrew D. Wilson Using a depth camera as a touch sensor , 2010, ITS '10.

[35]  Eisuke Fujinawa,et al.  Occlusion-aware Hand Posture Based Interaction on Tabletop Projector , 2019, UIST.

[36]  Buntarou Shizuki,et al.  Ray-Casting Based Interaction Using an Extended Pull-Out Gesture for Interactive Tabletops , 2019, OZCHI.

[37]  Enrico Costanza,et al.  TUIO: A Protocol for Table-Top Tangible User Interfaces , 2005 .

[38]  Enrico Rukzio,et al.  Pervasive information through constant personal projection: the ambient mobile pervasive display (AMP-D) , 2014, CHI.