Toolkit support for interactive projected displays

This paper presents a software toolkit designed to enable the rapid development of multimedia-rich, multi-touch enabled, and interactive projection-based displays. For instance: door displays, floor displays, wall displays, and interactive tables. Despite recent technological advances and the commercialization of hardware required to achieve this at a relatively low cost, creating and deploying such displays remains a difficult task, even for those with the essential technical skills and experience. We assert that greater accessibility of toolkits like the one presented in this paper will reduce these barriers and allow people (not necessarily from the ubiquitous computing domain) to apply the technology to their own fields. To assess this toolkit's suitability for this role, we present a system and usability evaluation. We observed that participants were able to quickly create their own novel display deployments. Our findings offer insights for potential toolkit users and those considering how to write programs for future ubiquitous projected display environments.

[1]  Joseph J. LaViola,et al.  Code bubbles: a working set-based interface for code understanding and maintenance , 2010, CHI.

[2]  Alessio Malizia,et al.  TESIS: turn every surface into an interactive surface , 2011, ITS '11.

[3]  Amy J. Ko,et al.  Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[4]  Myron W. Krueger,et al.  VIDEOPLACE—an artificial reality , 1985, CHI '85.

[5]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[6]  Jong-Il Park,et al.  Surface-Independent Direct-Projected Augmented Reality , 2006, ACCV.

[7]  Sebastian Boring,et al.  ProjectorKit: Easing the Development of Interactive Applications for Mobile Projectors , 2013 .

[8]  Albrecht Schmidt,et al.  Display Blindness: The Effect of Expectations on Attention towards Digital Signage , 2009, Pervasive.

[9]  Larry J. Hornbeck,et al.  Digital Light Processing and MEMS: Timely Convergence for a Bright Future , 2002 .

[10]  G. Toussaint Solving geometric problems with the rotating calipers , 1983 .

[11]  Liviu Iftode,et al.  Programming computers embedded in the physical world , 2003, The Ninth IEEE Workshop on Future Trends of Distributed Computing Systems, 2003. FTDCS 2003. Proceedings..

[12]  Arthur W. Combs,et al.  Individual Behavior: A New Frame of Reference for Psychology. , 1950 .

[13]  Xiang Cao,et al.  Multi-user interaction using handheld projectors , 2007, UIST.

[14]  Jürgen Steimle,et al.  FoldMe: interacting with double-sided foldable displays , 2012, Tangible and Embedded Interaction.

[15]  Andrew D. Wilson Using a depth camera as a touch sensor , 2010, ITS '10.

[16]  Greg Welch,et al.  Spatially augmented reality , 1999 .

[17]  Sidney Fels,et al.  Developing Cross-Display Applications Using the Really Easy Displays (RED) Framework , 2013 .

[18]  Jong-Il Park,et al.  Undistorted Projection onto Dynamic Surface , 2006, PSIVT.

[19]  Jan O. Borchers,et al.  Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at Public Displays? , 2008, Pervasive.

[20]  E. Rogers,et al.  Diffusion of Innovations , 1964 .

[21]  Jerry Alan Fails,et al.  Light widgets: interacting in every-day spaces , 2002, IUI '02.

[22]  Shumin Zhai,et al.  The performance of touch screen soft buttons , 2009, CHI.

[23]  Knut H. Sørensen,et al.  Making Technology Our Own?: Domesticating Technology Into Everyday Life , 1996 .

[24]  Claudio S. Pinhanez,et al.  Interacting with steerable projected displays , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[25]  Andreas Butz,et al.  Interactions in the air: adding further depth to interactive tabletops , 2009, UIST '09.

[26]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[27]  Hiroshi Ishii,et al.  Mechanical constraints as computational constraints in tabletop tangible interfaces , 2007, CHI.

[28]  A. Buryanov,et al.  Proportions of Hand Segments , 2010 .

[29]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[30]  Rahul Sukthankar,et al.  Smarter Presentations: Exploiting Homography in Camera-Projector Systems , 2001, ICCV.

[31]  Tony P. Pridmore,et al.  Pressing the Flesh: Sensing Multiple Touch and Finger Pressure on Arbitrary Surfaces , 2009, Pervasive.

[32]  Jörgen Sandberg How Do We Justify Knowledge Produced Within Interpretive Approaches? , 2005 .

[33]  Paul A. Beardsley,et al.  Natural video matting using camera arrays , 2006, ACM Trans. Graph..

[34]  James M. Rehg,et al.  Shadow Elimination and Blinding Light Suppression for Interactive Projected Displays , 2007, IEEE Trans. Vis. Comput. Graph..

[35]  Ivan Poupyrev,et al.  SideBySide: ad-hoc multi-user interaction with handheld projectors , 2011, UIST.

[36]  Jon Whittle,et al.  Digitally annexing desk space for software development: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[37]  Karsten Nebe,et al.  dSensingNI: a framework for advanced tangible interaction using a depth camera , 2012, TEI.

[38]  JeroenvanBaar RameshRaskar,et al.  A Low-Cost Projector Mosaic with Fast Registration , 2002 .

[39]  Peter Robinson,et al.  T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications , 2007, Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07).

[40]  Helen J. Wang,et al.  SurroundWeb: Least Privilege for Immersive "Web Rooms" , 2014 .

[41]  Stephanie D. Teasley,et al.  Rapid Software Development through Team Collocation , 2002, IEEE Trans. Software Eng..

[42]  Oliver Bimber,et al.  Real-Time Adaptive Radiometric Compensation , 2006, IEEE Transactions on Visualization and Computer Graphics.

[43]  Konrad Tollmar,et al.  Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges , 2008, NordiCHI 2008.

[44]  John Hardy Reflections: a year spent with an interactive desk , 2012, INTR.

[45]  Markus Funk,et al.  An augmented workplace for enabling user-defined tangibles , 2014, CHI Extended Abstracts.

[46]  Jonathan Grudin,et al.  Partitioning digital worlds: focal and peripheral awareness in multiple monitor use , 2001, CHI.

[47]  Gregory D. Abowd,et al.  What next, ubicomp?: celebrating an intellectual disappearing act , 2012, UbiComp.

[48]  Scott E. Hudson,et al.  Foldable interactive displays , 2008, UIST '08.

[49]  Andreas Butz,et al.  Curve: revisiting the digital desk , 2010, NordiCHI.

[50]  Joseph J. LaViola,et al.  Code bubbles: rethinking the user interface paradigm of integrated development environments , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[51]  James Fogarty,et al.  OASIS : Creating Smart Objects with Dynamic Digital Behaviors , 2010 .

[52]  Jerome H. Saltzer,et al.  End-to-end arguments in system design , 1984, TOCS.

[53]  Leon Cruickshank,et al.  Studios in software engineering education: Towards an evaluable model , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[54]  Paul P. Maglio,et al.  On Distinguishing Epistemic from Pragmatic Action , 1994, Cogn. Sci..

[55]  Johnny Chung Lee,et al.  Hacking the Nintendo Wii Remote , 2008, IEEE Pervasive Computing.

[56]  Shumin Zhai,et al.  High precision touch screen interaction , 2003, CHI '03.

[57]  Mark Ashdown Personal projected displays , 2004 .

[58]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[59]  Arie van Deursen,et al.  Adinda: a knowledgeable, browser-based IDE , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[60]  Enrico Costanza,et al.  TUIO: A Protocol for Table-Top Tangible User Interfaces , 2005 .

[61]  David Molyneaux,et al.  Smart object, not smart environment : cooperative augmentation of smart objects using projector-camera systems , 2008 .

[62]  Marisa S. Viveros,et al.  BlueSpace: personalizing workspace through awareness and adaptability , 2002, Int. J. Hum. Comput. Stud..

[63]  Nigel Davies,et al.  Using bluetooth device names to support interaction in smart environments , 2009, MobiSys '09.

[64]  Susan Leigh Star,et al.  Institutional Ecology, `Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39 , 1989 .

[65]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[66]  Maureen C. Stone Color and Brightness Appearance Issues in Tiled Displays , 2001, IEEE Computer Graphics and Applications.

[67]  Gerd Kortuem,et al.  Ubiquitous displays in dynamic environments: issues and opportunities , 2004 .

[68]  Eric von Hippel,et al.  Satisfying Heterogeneous User Needs Via Innovation Toolkits: The Case of Apache Security Software , 2002 .

[69]  Pattie Maes,et al.  WUW - wear Ur world: a wearable gestural interface , 2009, CHI Extended Abstracts.

[70]  Claudio S. Pinhanez The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces , 2001, UbiComp.

[71]  James Fogarty,et al.  Examining interaction with general-purpose object recognition in LEGO OASIS , 2011, 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[72]  Joshua Noble Programming Interactivity - A Designer's Guide to Processing, Arduino, and openFrameworks , 2009 .

[73]  Stacey D. Scott,et al.  Contextual design considerations for co-located, collaborative tables , 2008, 2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer Systems.

[74]  Martina Ziefle,et al.  Device-free interaction in smart domestic environments , 2013, AH.

[75]  Hiroshi Ishii,et al.  Emancipated pixels: real-world graphics in the luminous room , 1999, SIGGRAPH.

[76]  Matt Jones,et al.  Pico projectors: firefly or bright future? , 2012, INTR.

[77]  Eric Paulos,et al.  Rise of the expert amateur: DIY projects, communities, and cultures , 2010, NordiCHI.

[78]  Dominik Schmidt Know Thy Toucher , 2009 .

[79]  Ben Frain Responsive Web Design with HTML5 and CSS3 , 2012 .

[80]  Roland Siegwart,et al.  BRISK: Binary Robust invariant scalable keypoints , 2011, 2011 International Conference on Computer Vision.

[81]  Dominik Schmidt PhoneTouch: a technique for direct phone interaction on surfaces , 2010, ITS '10.

[82]  Claudio S. Pinhanez,et al.  Dynamically Reconfigurable Vision-Based User Interfaces , 2003, ICVS.

[83]  Chris Harrison,et al.  OmniTouch: wearable multitouch interaction everywhere , 2011, UIST.

[84]  Albrecht Schmidt,et al.  A New Era for Ubicomp Development , 2012, IEEE Pervasive Computing.

[85]  Daniel Vogel,et al.  Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users , 2004, UIST '04.

[86]  Mary Czerwinski,et al.  Code Thumbnails: Using Spatial Memory to Navigate Source Code , 2006, Visual Languages and Human-Centric Computing (VL/HCC'06).

[87]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[88]  Xiang Cao,et al.  Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces , 2012, Pervasive.

[89]  Andy Hopper,et al.  The active badge location system , 1992, TOIS.

[90]  Andreas Butz,et al.  Touch projector: mobile interaction through video , 2010, CHI.

[91]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[92]  Meredith Ringel Morris,et al.  Insights on Interactive Tabletops: A Survey of Researchers and Developers , 2009 .

[93]  Max Mühlhäuser,et al.  LightBeam: nomadic pico projector interaction with real world objects , 2012, CHI EA '12.

[94]  Oliver Bimber,et al.  Embedded entertainment with smart projectors , 2005, Computer.

[95]  Claudio S. Pinhanez Creating ubiquitous interactive games using everywhere display projectors , 2002, IWEC.

[96]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[97]  Claudio S. Pinhanez,et al.  To Frame or Not to Frame: The Role and Design of Frameless Displays in Ubiquitous Applications , 2005, UbiComp.

[98]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[99]  Robert DeLine,et al.  Code canvas: zooming towards better development environments , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[100]  Gudrun Klinker,et al.  KinectTouch: accuracy test for a very low-cost 2.5D multitouch tracking system , 2011, ITS '11.

[101]  Pieter Jan Stappers,et al.  Demonstrating generation Y interactions through interactive prototyping , 2011, UbiComp '11.

[102]  Panayiotis Zaphiris,et al.  Reading Online or on Paper: Which is Faster? , 2001 .

[103]  Florian Alt,et al.  Looking glass: a field study on noticing interactivity of a shop window , 2012, CHI.

[104]  Lloyd H. Nakatani,et al.  Soft machines: A philosophy of user-computer interface design , 1983, CHI '83.

[105]  B. Shneiderman,et al.  Improving the accuracy of touch screens: an experimental evaluation of three strategies , 1988, CHI '88.

[106]  Regan L. Mandryk,et al.  System Guidelines for Co-located, Collaborative Work on a Tabletop Display , 2003, ECSCW.