Application and validation of capacitive proximity sensing systems in smart environments

Smart environments feature a number of computing and sensing devices that support occupants in performing their tasks. In the last decades there has been a multitude of advances in miniaturizing sensors and computers, while greatly increasing their performance. As a result new devices are introduced into our daily lives that have a plethora of functions. Gathering information about the occupants is fundamental in adapting the smart environment according to preference and situation. There is a large number of different sensing devices available that can provide information about the user. They include cameras, accelerometers, GPS, acoustic systems, or capacitive sensors. The latter use the properties of an electric field to sense presence and properties of conductive objects within range. They are commonly employed in finger-controlled touch screens that are present in billions of devices. A less common variety is the capacitive proximity sensor. It can detect the presence of the human body over a distance, providing interesting applications in smart environments. Choosing the right sensor technology is an important decision in designing a smart environment application. Apart from looking at previous use cases, this process can be supported by providing more formal methods. In this work I present a benchmarking model that is designed to support this decision process for applications in smart environments. Previous benchmarks for pervasive systems have been adapted towards sensors systems and include metrics that are specific for smart environments. Based on distinct sensor characteristics, different ratings are used as weighting factors in calculating a benchmarking score. The method is verified using popularity matching in two scientific databases. Additionally, there are extensions to cope with central tendency bias and normalization with regards to average feature rating. Four relevant application areas are identified by applying this benchmark to applications in smart environments and capacitive proximity sensors. They are indoor localization, smart appliances, physiological sensing and gesture interaction. Any application area has a set of challenges regarding the required sensor technology, layout of the systems, and processing that can be tackled using various new or improved methods. I will present a collection of existing and novel methods that support processing data generated by capacitive proximity sensors. These are in the areas of sparsely distributed sensors, model-driven fitting methods, heterogeneous sensor systems, image-based processing and physiological signal processing. To evaluate the feasibility of these methods, several prototypes have been created and tested for performance and usability. Six of them are presented in detail. Based on these evaluations and the knowledge generated in the design process, I am able to classify capacitive proximity sensing in smart environments. This classification consists of a comparison to other popular sensing technologies in smart environments, the major benefits of capacitive proximity sensors, and their limitations. In order to support parties interested in developing smart environment applications using capacitive proximity sensors, I present a set of guidelines that support the decision process from technology selection to choice of processing methods.

[1]  Andreas Braun,et al.  Honeyfish-a high resolution gesture recognition system based on capacitive proximity sensing , 2012 .

[2]  Arjan Kuijper,et al.  Towards interactive car interiors: the active armrest , 2014, NordiCHI.

[3]  Daniel Jackson,et al.  FoodBoard: surface contact imaging for food recognition , 2013, UbiComp.

[4]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[5]  Khai N. Truong,et al.  BlueTone: a framework for interacting with public displays using dual-tone multi-frequency through bluetooth , 2009, UbiComp.

[6]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[7]  Arjan Kuijper,et al.  OpenCapSense: A rapid prototyping toolkit for pervasive interaction using capacitive sensing , 2013, 2013 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[8]  Andrew Wilson,et al.  Pointing in Intelligent Environments with the WorldCursor , 2003, INTERACT.

[9]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[10]  Lars Bretzner,et al.  Using marking menus to develop command sets for computer vision based hand gesture interfaces , 2002, NordiCHI '02.

[11]  Zicheng Liu,et al.  HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[13]  Andreas Braun,et al.  Designing a multi-purpose capacitive proximity sensing input device , 2011, PETRA '11.

[14]  Paul Lukowicz,et al.  Resonant magnetic coupling indoor localization system , 2013, UbiComp.

[15]  Gary L. Barrett Projected Capacitive Touch Screens , 2010 .

[16]  Tom G. Zimmerman,et al.  A hand gesture interface device , 1987, CHI '87.

[17]  Ivan Poupyrev,et al.  Botanicus Interacticus: interactive plants technology , 2012, SIGGRAPH '12.

[18]  Albrecht Schmidt,et al.  A Capacitive Sensing Toolkit for Pervasive Activity Detection and Recognition , 2007, Fifth Annual IEEE International Conference on Pervasive Computing and Communications (PerCom'07).

[19]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[20]  Pascal Getreuer,et al.  Linear Methods for Image Interpolation , 2011, Image Process. Line.

[21]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[22]  Joseph A. Paradiso,et al.  Electric Field Sensing For Graphical Interfaces , 1998, IEEE Computer Graphics and Applications.

[23]  Joseph A. Paradiso,et al.  Applying electric field sensing to human-computer interfaces , 1995, CHI '95.

[24]  Gregory D. Abowd,et al.  Instant inkjet circuits: lab-based inkjet printing to support rapid prototyping of UbiComp devices , 2013, UbiComp.

[25]  Tobias Alexander Große-Puppendahl,et al.  Enhancing Accelerometer-Based Activity Recognition with Capacitive Proximity Sensing , 2012, AmI.

[26]  Ondrej Krejcar,et al.  Use of Mobile Phones as Intelligent Sensors for Sound Input Analysis and Sleep State Detection , 2011, Sensors.

[27]  Jukka Vanhala,et al.  Unobtrusive human height and posture recognition with a capacitive sensor , 2011, Journal of Ambient Intelligence and Smart Environments.

[28]  J. M. Kim,et al.  Transparent conductive film based on carbon nanotubes and PEDOT composites , 2005 .

[29]  Brian P. Bailey,et al.  Towards a pervasive computing benchmark , 2005, Third IEEE International Conference on Pervasive Computing and Communications Workshops.

[30]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Bart Selman,et al.  Human Activity Detection from RGBD Images , 2011, Plan, Activity, and Intent Recognition.

[32]  Takayuki Kanda,et al.  Simultaneous people tracking and localization for social robots using external laser range finders , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  Andreas Braun Kontexterkennung über kapazitive Sensorarrays in Betten Context recognition using capacitive sensor arrays in beds , 2012 .

[34]  Massimo Bertozzi,et al.  GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection , 1998, IEEE Trans. Image Process..

[35]  Andreas Braun,et al.  Capacitive sensor-based hand gesture recognition in ambient intelligence scenarios , 2013, PETRA '13.

[36]  Masayuki Murata,et al.  Indoor Localization System using RSSI Measurement of Wireless Sensor Network based on ZigBee Standard , 2006, Wireless and Optical Communications.

[37]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[38]  Stefan Poslad,et al.  Ubiquitous Computing: Smart Devices, Environments and Interactions , 2009 .

[39]  Kelly DeRango,et al.  The effects of an office ergonomics training and chair intervention on worker knowledge, behavior and musculoskeletal risk. , 2009, Applied ergonomics.

[40]  Steven A. Shafer,et al.  XWand: UI for intelligent spaces , 2003, CHI '03.

[41]  Paul Coulton,et al.  Using a Mobile Phone as a "Wii-like" Controller for Playing Games on a Large Public Display , 2008, Int. J. Comput. Games Technol..

[42]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[43]  Alessandro Saffiotti,et al.  PEIS Ecology: integrating robots into smart environments , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[44]  Bryon C. Lewis,et al.  The Evolution of Benchmarking as a Computer Performance Evaluation Technique , 1985, MIS Q..

[45]  Andreas Braun,et al.  AmbiTrack - Marker-free Indoor Localization and Tracking of Multiple Users in Smart Environments with a Camera-based Approach , 2013, EvAAL.

[46]  Neal Patwari,et al.  Radio Tomographic Imaging with Wireless Networks , 2010, IEEE Transactions on Mobile Computing.

[47]  William Ribarsky,et al.  Speech and Gesture Multimodal Control of a Whole Earth 3D Visualization Environment , 2002, VisSym.

[48]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[49]  James E. Smith,et al.  Characterizing computer performance with a single number , 1988, CACM.

[50]  K. Wehrle,et al.  Accurate prediction of power consumption in sensor networks , 2005, The Second IEEE Workshop on Embedded Networked Sensors, 2005. EmNetS-II..

[51]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[52]  V. Jamalabad,et al.  Self-organization of a Heterogeneous Sensor Network by Genetic Algorithms , 1998 .

[53]  Arjan Kuijper,et al.  Swiss-cheese extended: an object recognition method for ubiquitous interfaces based on capacitive proximity sensing , 2013, CHI.

[54]  Ali A. Nazari Shirehjini,et al.  A novel interaction metaphor for personal environment control: direct manipulation of physical environment based on 3D visualization , 2004, Comput. Graph..

[55]  Miika Valtonen,et al.  Capacitive User Tracking Methods for Smart Environments , 2012 .

[56]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[57]  Esther Rodríguez-Villegas,et al.  Breathing Detection: Towards a Miniaturized, Wearable, Battery-Operated Monitoring System , 2008, IEEE Transactions on Biomedical Engineering.

[58]  Jane Yung-jen Hsu,et al.  The Diet-Aware Dining Table: Observing Dietary Behaviors over a Tabletop Surface , 2006, Pervasive.

[59]  Leena Ventä-Olkkonen,et al.  User evaluation of mobile augmented reality scenarios , 2012, J. Ambient Intell. Smart Environ..

[60]  Arjan Kuijper,et al.  CapTap: combining capacitive gesture recognition and acoustic touch detection , 2016, iWOAR.

[61]  Kai Keng Ang,et al.  EEG-based Emotion Recognition Using Self-Organizing Map for Boundary Detection , 2010, 2010 20th International Conference on Pattern Recognition.

[62]  Bilge Mutlu,et al.  MACH: my automated conversation coach , 2013, UbiComp.

[63]  Joshua R. Smith Field Mice: Extracting Hand Geometry from Electric Field Measurements , 1996, IBM Syst. J..

[64]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[65]  Andreas Braun,et al.  Marker-Free Indoor Localization and Tracking of Multiple Users in Smart Environments Using a Camera-Based Approach , 2013, HCI.

[66]  Chris Harrison,et al.  Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces , 2008, UIST '08.

[67]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.

[68]  Sebastian Boring,et al.  HandSense: discriminating different ways of grasping and holding a tangible user interface , 2009, Tangible and Embedded Interaction.

[69]  Jack J. Dongarra,et al.  The LINPACK Benchmark: past, present and future , 2003, Concurr. Comput. Pract. Exp..

[70]  Yang Li,et al.  Experimental analysis of touch-screen gesture designs in mobile environments , 2011, CHI.

[71]  Ivan Poupyrev,et al.  Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects , 2012, CHI.

[72]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[73]  Claire A. G. J. Huijnen,et al.  "Maybe It Becomes a Buddy, But Do Not Call It a Robot" - Seamless Cooperation between Companion Robotics and Smart Homes , 2011, AmI.

[74]  Joshua R. Smith,et al.  Electric field imaging , 1999 .

[75]  Simon A. Dobson,et al.  Situation identification techniques in pervasive computing: A review , 2012, Pervasive Mob. Comput..

[76]  Alessandro Saffiotti,et al.  Symbiotic Robotic Systems: Humans, Robots, and Smart Environments , 2006, IEEE Intelligent Systems.

[77]  Ken Hinckley,et al.  A survey of design issues in spatial input , 1994, UIST '94.

[78]  Hamid Aghajan,et al.  Smart home care network using sensor fusion and distributed vision-based reasoning , 2006, VSSN '06.

[79]  Ivan Poupyrev,et al.  Capacitive fingerprinting: exploring user differentiation by sensing electrical properties of the human body , 2012, UIST '12.

[80]  Galen Panger Kinect in the kitchen: testing depth camera interactions in practical home environments , 2012, CHI EA '12.

[81]  J. Huttenlocher,et al.  Category Effects on Estimates of Stimuli: Perception or Reconstruction? , 2000, Psychological science.

[82]  Andreas Braun,et al.  CapFloor - A Flexible Capacitive Indoor Localization System , 2011, EvAAL.

[83]  Arjan Kuijper,et al.  Capacitive proximity sensing in smart environments , 2015, J. Ambient Intell. Smart Environ..

[84]  Arjan Kuijper,et al.  A Benchmarking Model for Sensors in Smart Environments , 2014, AmI.

[85]  JungHyun Han,et al.  Manufacturing feature recognition from solid models: a status report , 2000, IEEE Trans. Robotics Autom..

[86]  Sang Hyun Park,et al.  Smart home – digitally engineered domestic life , 2003, Personal and Ubiquitous Computing.

[87]  Alanson P. Sample,et al.  Wirelessly powered bistable display tags , 2013, UbiComp.

[88]  Jun Rekimoto,et al.  Three-Dimensional Mid-Air Acoustic Manipulation by Ultrasonic Phased Arrays , 2013, PloS one.

[89]  Bill N. Schilit,et al.  Context-aware computing applications , 1994, Workshop on Mobile Computing Systems and Applications.

[90]  Michael Beigl,et al.  The MediaCup: Awareness Technology Embedded in a Everyday Object , 1999, HUC.

[91]  O. Wade,et al.  Movements of the thoracic cage and diaphragm in respiration * , 1954, The Journal of physiology.

[92]  Andreas Braun,et al.  MoviBed - Sleep Analysis Using Capacitive Sensors , 2014, HCI.

[93]  Tobias Alexander Große-Puppendahl,et al.  Classification of User Postures with Capacitive Proximity Sensors in AAL-Environments , 2011, AmI.

[94]  Michael Rohs,et al.  The smart phone: a ubiquitous input device , 2006, IEEE Pervasive Computing.

[95]  Paolo Fiorini,et al.  A robotics wheelchair for crowded public environment , 2001, IEEE Robotics Autom. Mag..

[96]  Jukka Vanhala,et al.  TileTrack: Capacitive human tracking using floor tiles , 2009, 2009 IEEE International Conference on Pervasive Computing and Communications.

[97]  T Salmi,et al.  Automatic analysis of sleep records with static charge sensitive bed. , 1986, Electroencephalography and clinical neurophysiology.

[98]  Chris Schmandt,et al.  AudioStreamer: exploiting simultaneity for listening , 1995, CHI 95 Conference Companion.

[99]  Suranga Nanayakkara,et al.  StickEar: making everyday objects respond to sound , 2013, UIST.

[100]  Jefferson Y. Han Low-cost multi-touch sensing through frustrated total internal reflection , 2005, UIST.

[101]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[102]  Tovi Grossman,et al.  Medusa: a proximity-aware multi-touch tabletop , 2011, UIST.

[103]  Andreas Butz,et al.  How to make large touch screens usable while driving , 2013, AutomotiveUI.

[104]  Jun Li,et al.  Crowd++: unsupervised speaker count with smartphones , 2013, UbiComp.

[105]  Ray Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[106]  Jukka Vanhala,et al.  Human body tracking with electric field ranging , 2010, MindTrek.

[107]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[108]  Venkata N. Padmanabhan,et al.  Indoor localization without the pain , 2010, MobiCom.

[109]  Yuanqing Xia,et al.  Multi-Sensor Data Fusion , 2014 .

[110]  Norbert Wehn,et al.  Performance evaluation of ambient services by combining robotic frameworks and a smart environment platform , 2013, Robotics Auton. Syst..

[111]  Tomomasa Sato,et al.  Human motion tracking system based on skeleton and surface integration model using pressure sensors distribution bed , 2000, Proceedings Workshop on Human Motion.

[112]  Reinhold Weicker,et al.  Dhrystone: a synthetic systems programming benchmark , 1984, CACM.

[113]  Tsutomu Terada,et al.  Ultrasound-based movement sensing, gesture-, and context-recognition , 2013, ISWC '13.

[114]  Hari Balakrishnan,et al.  6th ACM/IEEE International Conference on on Mobile Computing and Networking (ACM MOBICOM ’00) The Cricket Location-Support System , 2022 .

[115]  Bharti Bansal,et al.  Gesture Recognition: A Survey , 2016 .

[116]  Gary Barrett,et al.  Projected‐Capacitive Touch Technology , 2010 .

[117]  Alessandro Valli,et al.  The design of natural interaction , 2008, Multimedia Tools and Applications.

[118]  Henry A. Kautz,et al.  Real-time crowd labeling for deployable activity recognition , 2013, CSCW.

[119]  Mathieu Le Goc,et al.  A low-cost transparent electric field sensor for 3d interaction on mobile devices , 2014, CHI.

[120]  Andreas Braun,et al.  The Capacitive Chair , 2015, HCI.

[121]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[122]  Patrick Baudisch,et al.  Modular and deformable touch-sensitive surfaces based on time domain reflectometry , 2011, UIST.

[123]  Alex Pentland,et al.  Face Recognition for Smart Environments , 2000, Computer.

[124]  Alex Pentland,et al.  A sensing chair using pressure distribution sensors , 2001 .

[125]  J. L. Roux An Introduction to the Kalman Filter , 2003 .

[126]  Chris Harrison,et al.  TapSense: enhancing finger interaction on touch surfaces , 2011, UIST.

[127]  Paul Lukowicz,et al.  Active Capacitive Sensing: Exploring a New Wearable Sensing Modality for Activity Recognition , 2010, Pervasive.

[128]  Boby George,et al.  Seat Occupancy Detection Based on Capacitive Sensing , 2009, IEEE Transactions on Instrumentation and Measurement.

[129]  Diane J. Cook,et al.  How smart are our environments? An updated look at the state of the art , 2007, Pervasive Mob. Comput..

[130]  Stefano Chessa,et al.  Evaluating AAL Systems Through Competitive Benchmarking. Indoor Localization and Tracking , 2011, Communications in Computer and Information Science.

[131]  Sriram Subramanian,et al.  Multi-layer interaction for digital tables , 2006, UIST.

[132]  Mike Wu,et al.  Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays , 2003, UIST '03.

[133]  Kristof Van Laerhoven,et al.  How to build smart appliances? , 2001, IEEE Personal Communications.

[134]  H. Schulz,et al.  Rate and Distribution of Body Movements during Sleep in Humans , 1983, Perceptual and motor skills.

[135]  Chi-Sheng Shih,et al.  Smart Medication Dispenser: Design, Architecture and Implementation , 2011, IEEE Systems Journal.

[136]  Albrecht Schmidt,et al.  Thracker - Using Capacitive Sensing for Gesture Recognition , 2006, 26th IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW'06).

[137]  Björn W. Schuller,et al.  Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling , 2010, INTERSPEECH.

[138]  Zhengyou Zhang,et al.  Maximum Likelihood Sound Source Localization and Beamforming for Directional Microphone Arrays in Distributed Meetings , 2008, IEEE Transactions on Multimedia.

[139]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[140]  Dieter Schmalstieg,et al.  Indoor Positioning and Navigation with Camera Phones , 2009, IEEE Pervasive Computing.

[141]  Arjan Kuijper,et al.  Providing Visual Support for Selecting Reactive Elements in Intelligent Environments , 2013, Trans. Comput. Sci..