Wall++: Room-Scale Interactive and Context-Aware Sensing

Human environments are typified by walls, homes, offices, schools, museums, hospitals and pretty much every indoor context one can imagine has walls. In many cases, they make up a majority of readily accessible indoor surface area, and yet they are static their primary function is to be a wall, separating spaces and hiding infrastructure. We present Wall++, a low-cost sensing approach that allows walls to become a smart infrastructure. Instead of merely separating spaces, walls can now enhance rooms with sensing and interactivity. Our wall treatment and sensing hardware can track users' touch and gestures, as well as estimate body pose if they are close. By capturing airborne electromagnetic noise, we can also detect what appliances are active and where they are located. Through a series of evaluations, we demonstrate Wall++ can enable robust room-scale interactive and context-aware applications.

[1]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Jun Rekimoto,et al.  Mirage: exploring interaction modalities using off-body static electric field sensing , 2013, UIST.

[3]  Amarjeet Singh,et al.  An in depth study into using EMI signatures for appliance identification , 2014, BuildSys@SenSys.

[4]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[5]  Luciano Lavagno,et al.  A Tagless Indoor Localization System Based on Capacitive Sensing Technology , 2016, Sensors.

[6]  Mohammad Albaida Object-based Activity Recognition with Heterogeneous Sensors on Wrist , 2018 .

[7]  Jing Liu,et al.  Survey of Wireless Indoor Positioning Techniques and Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[8]  Kevin Curran,et al.  A survey of active and passive indoor localisation systems , 2012, Comput. Commun..

[9]  Jukka Vanhala,et al.  Unobtrusive human height and posture recognition with a capacitive sensor , 2011, Journal of Ambient Intelligence and Smart Environments.

[10]  J. Bartlett,et al.  Product information , 2001, Transplantation.

[11]  H.T. Friis,et al.  A Note on a Simple Transmission Formula , 1946, Proceedings of the IRE.

[12]  Jun Rekimoto,et al.  SmartSkin: an infrastructure for freehand manipulation on interactive surfaces , 2002, CHI.

[13]  Neil Gershenfeld,et al.  An installation of interactive furniture , 2000, IBM Syst. J..

[14]  Gierad Laput,et al.  Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances , 2017, CHI.

[15]  Michael Rohs,et al.  USING CAMERA-EQUIPPED MOBILE PHONES FOR INTERACTING WITH REAL-WORLD OBJECTS , 2004 .

[16]  Jukka Vanhala,et al.  TileTrack: Capacitive human tracking using floor tiles , 2009, 2009 IEEE International Conference on Pervasive Computing and Communications.

[17]  Timo Hämäläinen,et al.  Experiments on local positioning with Bluetooth , 2003, Proceedings ITCC 2003. International Conference on Information Technology: Coding and Computing.

[18]  Shwetak N. Patel,et al.  DoppleSleep: a contactless unobtrusive sleep sensing system using short-range Doppler radar , 2015, UbiComp.

[19]  Alanson P. Sample,et al.  RapID: A Framework for Fabricating Low-Latency Interactive Objects with RFID Tags , 2016, CHI.

[20]  Joseph A. Paradiso,et al.  Leveraging conductive inkjet technology to build a scalable and versatile surface for ubiquitous sensing , 2011, UbiComp '11.

[21]  Dina Katabi,et al.  RF-IDraw: virtual touch screen in the air using RF signals , 2014, S3 '14.

[22]  Joseph A. Paradiso,et al.  PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play , 1999, CHI '99.

[23]  Joseph A. Paradiso,et al.  Sensor systems for interactive surfaces , 2000, IBM Syst. J..

[24]  Joseph A. Paradiso,et al.  EMI Spy: Harnessing electromagnetic interference for low-cost, rapid prototyping of proxemic interaction , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[25]  Yunhao Liu,et al.  LANDMARC: Indoor Location Sensing Using Active RFID , 2004, Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)..

[26]  Anthony Rowe,et al.  Contactless sensing of appliance state transitions through variations in electromagnetic fields , 2010, BuildSys '10.

[27]  Gierad Laput,et al.  Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds , 2015, CHI.

[28]  Moustafa Youssef,et al.  CoSDEO 2016 Keynote: A decade later — Challenges: Device-free passive localization for wireless environments , 2007, 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[29]  Gierad Laput,et al.  Electrick: Low-Cost Touch Sensing Using Electric Field Tomography , 2017, CHI.

[30]  Xiang 'Anthony' Chen,et al.  Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions , 2016, CHI.

[31]  Joseph A. Paradiso,et al.  Passive acoustic knock tracking for interactive windows , 2002, CHI Extended Abstracts.

[32]  Joshua R. Smith,et al.  Electric field imaging , 1999 .

[33]  Chris Harrison,et al.  Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces , 2008, UIST '08.

[34]  Arjan Kuijper,et al.  CapTap: combining capacitive gesture recognition and acoustic touch detection , 2016, iWOAR.

[35]  Otmar Hilliges,et al.  Steerable augmented reality with the beamatron , 2012, UIST.

[36]  Tobias Alexander Große-Puppendahl,et al.  Classification of User Postures with Capacitive Proximity Sensors in AAL-Environments , 2011, AmI.

[37]  Widyawan,et al.  Indoor human tracking application using multiple depth-cameras , 2012, 2012 International Conference on Advanced Computer Science and Information Systems (ICACSIS).

[38]  Joseph A. Paradiso,et al.  Cost-effective wearable sensor to detect EMF , 2009, CHI Extended Abstracts.

[39]  Shwetak N. Patel,et al.  ID-Match: A Hybrid Computer Vision and RFID System for Recognizing Individuals in Groups , 2016, CHI.

[40]  Neal Patwari,et al.  Radio Tomographic Imaging with Wireless Networks , 2010, IEEE Transactions on Mobile Computing.

[41]  Mani B. Srivastava,et al.  ViridiScope: design and implementation of a fine grained power monitoring system for homes , 2009, UbiComp.

[42]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[43]  Andruid Kerne,et al.  ZeroTouch: an optical multi-touch and free-air interaction architecture , 2012, CHI.

[44]  Anthony Rowe,et al.  A magnetic field-based appliance metering system , 2013, 2013 ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS).

[45]  Eric C. Larson,et al.  HeatWave: thermal imaging for surface user interaction , 2011, CHI.

[46]  Joseph A. Paradiso,et al.  Electric Field Sensing For Graphical Interfaces , 1998, IEEE Computer Graphics and Applications.

[47]  Joseph A. Paradiso,et al.  Applying electric field sensing to human-computer interfaces , 1995, CHI '95.

[48]  Shwetak N. Patel,et al.  ElectriSense: single-point sensing using EMI for electrical event detection and classification in the home , 2010, UbiComp.

[49]  G. W. Hart,et al.  Nonintrusive appliance load monitoring , 1992, Proc. IEEE.

[50]  J. Rekimoto,et al.  Perceptual Surfaces : Towards a Human and Object Sensitive Interactive Display , 1997 .

[51]  Eric Paulos,et al.  UpStream: motivating water conservation with low-cost water flow sensing and persuasive displays , 2010, CHI.

[52]  Patrick Baudisch,et al.  Multitoe: high-precision interaction with back-projected floors based on high-resolution multi-touch input , 2010, UIST.

[53]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[54]  Robert A. Moog,et al.  Theremin: Ether Music and Espionage , 2000 .

[55]  Pao-Chi Chang,et al.  People tracking in an environment with multiple depth cameras: A skeleton-based pairwise trajectory matching scheme , 2016, J. Vis. Commun. Image Represent..

[56]  Gierad Laput,et al.  EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects , 2015, UIST.

[57]  Gary Barrett,et al.  Projected‐Capacitive Touch Technology , 2010 .

[58]  Robert Xiao,et al.  Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation , 2014, MobileHCI '14.

[59]  Blair MacIntyre,et al.  RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units , 2014, UIST.

[60]  Frédo Durand,et al.  Capturing the human figure through a wall , 2015, ACM Trans. Graph..

[61]  Shwetak N. Patel,et al.  PaperID: A Technique for Drawing Functional Battery-Free Wireless Interfaces on Paper , 2016, CHI.

[62]  Gierad Laput,et al.  ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers , 2016, UIST.

[63]  R. King Electromagnetic waves and antennas above and below the surface of the earth , 1979 .

[64]  Matthew S. Reynolds,et al.  Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction , 2017, CHI.

[65]  Hrvoje Benko,et al.  Dyadic projected spatial augmented reality , 2014, UIST.

[66]  Jue Wang,et al.  RF-IDraw: virtual touch screen in the air using RF signals , 2015, SIGCOMM 2015.

[67]  Alanson P. Sample,et al.  IDSense: A Human Object Interaction Detection System Based on Passive UHF RFID , 2015, CHI.

[68]  Stoyan Nihtianov,et al.  for Capacitive Sensors , 2011 .

[69]  Alanson P. Sample,et al.  EM-ID: Tag-less identification of electrical devices via electromagnetic emissions , 2016, 2016 IEEE International Conference on RFID (RFID).

[70]  Arjan Kuijper,et al.  Capacitive near-field communication for ubiquitous interaction and perception , 2014, UbiComp.

[71]  Josef Hallberg,et al.  Positioning with Bluetooth , 2003, 10th International Conference on Telecommunications, 2003. ICT 2003..