Knock knock, what's there: converting passive objects into customizable smart controllers

Knocking is a way of interacting with everyday objects. We introduce BeatIt, a novel technique that allows users to use passive, everyday objects to control a smart environment by recognizing the sounds generated from knocking on the objects. BeatIt uses a BeatSet, a series of percussive sound samples, to represent the sound signature of knocking on an object. A user associates a BeatSet with an event. For example, a user can associate the BeatSet of knocking on a door with the event of turning on the lights. Decoder, a signal-processing module, classifies the sound signals into one of the recorded BeatSets, and then triggers the associated event. Unlike prior work, BeatIt can be implemented on microphone-enabled commodity devices. Our user studies with 12 participants showed that our proof-of-concept implementation based on a smartwatch could accurately classify eight BeatSets using a user-independent classifier.

[1]  Tsukasa Ogasawara,et al.  Active bone-conducted sound sensing for wearable interfaces , 2011, UIST '11 Adjunct.

[2]  Fulvio Corno,et al.  HomeRules: A Tangible End-User Programming Interface for Smart Homes , 2015, CHI Extended Abstracts.

[3]  Gregory D. Abowd,et al.  Whoosh: non-voice acoustics for low-cost, hands-free, and rapid input on smartwatches , 2016, SEMWEB.

[4]  Buntarou Shizuki,et al.  Touch & activate: adding interactivity to existing objects using active acoustic sensing , 2013, UIST.

[5]  Kimiya Yamaashi,et al.  Object-oriented video: interaction with real-world objects through live video , 1992, CHI.

[6]  Sebastian Möller,et al.  I'm home: Defining and evaluating a gesture set for smart-home control , 2011, Int. J. Hum. Comput. Stud..

[7]  Ferdinando Grossi,et al.  Light on! Real world evaluation of a P300-based brain–computer interface (BCI) for environment control in a smart home , 2012, Ergonomics.

[8]  Gierad Laput,et al.  Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds , 2015, CHI.

[9]  Anind K. Dey,et al.  a CAPpella: programming by demonstration of context-aware applications , 2004, CHI.

[10]  Takuya Maekawa,et al.  Recognizing the Use of Portable Electrical Devices with Hand-Worn Magnetic Sensors , 2011, Pervasive.

[11]  Gierad Laput,et al.  EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects , 2015, UIST.

[12]  Blase Ur,et al.  Practical trigger-action programming in the smart home , 2014, CHI.

[13]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[14]  David Wetherall,et al.  Recognizing daily activities with RFID-based sensors , 2009, UbiComp.

[15]  Xiang Cao,et al.  PICOntrol: using a handheld projector for direct control of physical devices through visible light , 2012, UIST.

[16]  Chris Harrison,et al.  Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces , 2008, UIST '08.

[17]  Gierad Laput,et al.  ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers , 2016, UIST.

[18]  Jacob O. Wobbrock,et al.  TapSongs: tapping rhythm-based passwords on a single binary sensor , 2009, UIST '09.

[19]  CLICK CLICK,et al.  SuperVision: Spatial Control of Connected Objects in a Smart Home , 2015 .

[20]  Kori Inkpen Quinn,et al.  That one there! Pointing to establish device identity , 2002, UIST '02.

[21]  Gregory D. Abowd,et al.  The Georgia Tech aware home , 2008, CHI Extended Abstracts.

[22]  Fulvio Corno,et al.  DOGeye: Controlling your home with eye interaction , 2011, Interact. Comput..

[23]  Günter Edlinger,et al.  A Hybrid Brain-Computer Interface for Smart Home Control , 2011, HCI.

[24]  Erin Walker,et al.  A tangible programming tool for creation of context-aware applications , 2013, UbiComp.

[25]  Desney S. Tan,et al.  Skinput: appropriating the body as an input surface , 2010, CHI.

[26]  Rainer Brunn.  The sound of one hand , 1995 .

[27]  Alain Dufaux.  Detection and Recognition of Impulsive Sound Signals , 2001 .

[28]  Daisuke Sakamoto,et al.  CRISTAL: a collaborative home media and device controller based on a multi-touch display , 2009, ITS '09.

[29]  Lei Shi,et al.  Tickers and Talker: An Accessible Labeling Toolkit for 3D Printed Models , 2016, CHI.

[30]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[31]  Donghun Lee,et al.  Towards successful user interaction with systems: focusing on user-derived gestures for smart home systems. , 2014, Applied ergonomics.

[32]  Timothy Sohn,et al.  iCAP: Interactive Prototyping of Context-Aware Applications , 2006, Pervasive.

[33]  Yi Zhao,et al.  A battery-free object localization and motion sensing platform , 2014, UbiComp.

[34]  Wei Pan,et al.  SoundSense: scalable sound sensing for people-centric applications on mobile phones , 2009, MobiSys '09.

[35]  Pedro Lopes,et al.  Augmenting touch interaction through acoustic sensing , 2011, ITS '11.

[36]  Gregory D. Abowd,et al.  CAMP: A Magnetic Poetry Interface for End-User Programming of Capture Applications for the Home , 2004, UbiComp.

[37]  Lei Shi.  Talkabel: A Labeling Method for 3D Printed Models , 2015, ASSETS.

[38]  Chris Harrison,et al.  TapSense: enhancing finger interaction on touch surfaces , 2011, UIST.

[39]  Gregory D. Abowd,et al.  Ubicomp 2001: Ubiquitous Computing , 2001, Lecture Notes in Computer Science.

[40]  Joshua R. Smith,et al.  RFID-based techniques for human-activity detection , 2005, Commun. ACM.

[41]  Sarah Mennicken,et al.  "It's like living with a friendly stranger": perceptions of personality traits in a smart home , 2016, UbiComp.

[42]  Gregory D. Abowd,et al.  A 2-Way Laser-Assisted Selection Scheme for Handhelds in a Physical Environment , 2003, UbiComp.

[43]  Desney S. Tan,et al.  Your noise is my command: sensing gestures using the body as an antenna , 2011, CHI.

[44]  Stefan Saroiu,et al.  Home automation in the wild: challenges and opportunities , 2011, CHI.

[45]  James Fogarty,et al.  Sensing from the basement: a feasibility study of unobtrusive and low-cost home activity recognition , 2006, UIST.

[46]  Marlien Herselman,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2015 .

[47]  Jo Vermeulen,et al.  From today's augmented houses to tomorrow's smart homes: new directions for home automation research , 2014, UbiComp.

[48]  John Seely Brown,et al.  The Origins of Ubiquitous Computing Research at PARC in the Late 1980s , 1999, IBM Syst. J..

[49]  Michael Schaefer,et al.  Effects of different viewing perspectives on somatosensory activations during observation of touch , 2009, Human brain mapping.

[50]  Gierad Laput,et al.  Electrick: Low-Cost Touch Sensing Using Electric Field Tomography , 2017, CHI.

[51]  Alex Pentland,et al.  Auditory Context Awareness via Wearable Computing , 1998 .

[52]  Matthai Philipose,et al.  Egocentric recognition of handled objects: Benchmark and analysis , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[53]  W. Keith Edwards,et al.  At Home with Ubiquitous Computing: Seven Challenges , 2001, UbiComp.

[54]  Takuya Maekawa,et al.  Recognizing Handheld Electrical Device Usage with Hand-Worn Coil of Wire , 2012, Pervasive.

[55]  Alanson P. Sample,et al.  IDSense: A Human Object Interaction Detection System Based on Passive UHF RFID , 2015, CHI.

[56]  Roderick Murray-Smith,et al.  Stane: synthesized surfaces for tactile input , 2008, CHI.

[57]  Arjan Kuijper,et al.  Capacitive near-field communication for ubiquitous interaction and perception , 2014, UbiComp.

[58]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Tack-Don Han,et al.  Ambient Wall: Smart Wall Display interface which can be controlled by simple gesture for smart home , 2011, SA '11.

[60]  Loren G. Terveen,et al.  The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface , 2002, CHI Extended Abstracts.

[61]  Stephen S. Intille,et al.  Designing a Home of the Future , 2002, IEEE Pervasive Comput..

[62]  Steve Hodges,et al.  Assessing and Optimizing the Range of UHF RFID to Enable Real-World Pervasive Computing Applications , 2007, Pervasive.