Characterizing Multidimensional Capacitive Servoing for Physical Human–Robot Interaction

Towards the goal of robots performing robust and intelligent physical interactions with people, it is crucial that robots are able to accurately sense the human body, follow trajectories around the body, and track human motion. This study introduces a capacitive servoing control scheme that allows a robot to sense and navigate around human limbs during close physical interactions. Capacitive servoing leverages temporal measurements from a multi-electrode capacitive sensor array mounted on a robot’s end effector to estimate the relative position and orientation (pose) of a nearby human limb. Capacitive servoing then uses these human pose estimates from a datadriven pose estimator within a feedback control loop in order to maneuver the robot’s end effector around the surface of a human limb. We provide a design overview of capacitive sensors for human-robot interaction and then investigate the performance and generalization of capacitive servoing through an experiment with 12 human participants. The results indicate that multidimensional capacitive servoing enables a robot’s end effector to move proximally or distally along human limbs while adapting to human pose. Using a cross-validation experiment, results further show that capacitive servoing generalizes well across people with different body size.

[1]  Satoshi Tsuji,et al.  Proximity Skin Sensor Using Time-of-Flight Sensor for Human Collaborative Robot , 2019, IEEE Sensors Journal.

[2]  Simona Crea,et al.  Controlling a Robotic Hip Exoskeleton With Noncontact Capacitive Sensors , 2019, IEEE/ASME Transactions on Mechatronics.

[3]  Gary Barrett,et al.  Projected‐Capacitive Touch Technology , 2010 .

[4]  Rajni V. Patel,et al.  Visual servoing in medical robotics: a survey. Part II: tomographic imaging modalities – techniques and applications , 2015, The international journal of medical robotics + computer assisted surgery : MRCAS.

[5]  Nicolas Andreff,et al.  Visual servoing controller for time-invariant 3D path following with remote centre of motion constraint , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[6]  W. Wilson,et al.  Comparison of Basic Visual Servoing Methods , 2011, IEEE/ASME Transactions on Mechatronics.

[7]  Nassir Navab,et al.  Human Pose Estimation from Pressure Sensor Data , 2018, Bildverarbeitung für die Medizin.

[8]  Carme Torras,et al.  Perception of cloth in assistive robotic manipulation tasks , 2020, Natural Computing.

[9]  Yiannis Demiris,et al.  Iterative path optimisation for personalised dressing assistance using vision and force information , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[10]  Greg Turk,et al.  Learning to Collaborate From Simulation for Robot-Assisted Dressing , 2019, IEEE Robotics and Automation Letters.

[11]  Ulrike Thomas,et al.  Collision Avoidance with Proximity Servoing for Redundant Serial Robot Manipulators , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Gernot Bahle,et al.  Designing Sensitive Wearable Capacitive Sensors for Activity Recognition , 2013, IEEE Sensors Journal.

[13]  N. Hollis,et al.  Prevalence of Disabilities and Health Care Access by Disability Status and Type Among Adults — United States, 2016 , 2018, MMWR. Morbidity and mortality weekly report.

[14]  Alois Knoll,et al.  Framework of automatic robot surgery system using Visual servoing , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  L. Beccai,et al.  Flexible Three‐Axial Force Sensor for Soft and Highly Sensitive Artificial Touch , 2014, Advanced materials.

[16]  Gamini Dissanayake,et al.  Capacitive sensor for object ranging and material type identification , 2008 .

[17]  Zheng Liu,et al.  Flexible Sensing Electronics for Wearable/Attachable Health Monitoring. , 2017, Small.

[18]  C. Karen Liu,et al.  Multidimensional Capacitive Sensing for Robot-Assisted Dressing and Bathing , 2019, 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR).

[19]  Christian Duriez,et al.  Proximity Perception in Human-Centered Robotics: A Survey on Sensing Systems and Applications , 2021, IEEE Transactions on Robotics.

[20]  Peter I. Corke,et al.  A tutorial on visual servo control , 1996, IEEE Trans. Robotics Autom..

[21]  Jürgen Beyerer,et al.  Visual Servoing , 2012, Autom..

[22]  Oussama Khatib,et al.  Capacitive skin sensors for robot impact monitoring , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[23]  Kimitoshi Yamazaki,et al.  Bottom dressing by a life-sized humanoid robot provided failure detection and recovery functions , 2014, 2014 IEEE/SICE International Symposium on System Integration.

[24]  Giorgio Metta,et al.  Methods and Technologies for the Implementation of Large-Scale Robot Tactile Sensors , 2011, IEEE Transactions on Robotics.

[25]  Greg Turk,et al.  Bodies at Rest: 3D Human Pose and Shape Estimation From a Pressure Image Using Synthetic Data , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Heinz Wörn,et al.  Methods for safe human-robot-interaction using capacitive tactile proximity sensors , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Oussama Khatib,et al.  Virtual whiskers — Highly responsive robot collision avoidance , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  C. Karen Liu,et al.  Personalized collaborative plans for robot-assisted dressing via optimization and simulation , 2019, Autonomous Robots.

[29]  Hubert Zangl,et al.  Gesture-based Contactless Control of Mobile Manipulators using Capacitive Sensing , 2020, 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[30]  Ulrike Thomas,et al.  With Proximity Servoing towards Safe Human-Robot-Interaction , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[31]  Nassir Navab,et al.  Electromagnetic Servoing—A New Tracking Paradigm , 2013, IEEE Transactions on Medical Imaging.

[32]  Luc Soler,et al.  Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing , 2003, IEEE Trans. Robotics Autom..

[33]  Joshua R. Smith,et al.  An Electric Field Pretouch system for grasping and co-manipulation , 2010, 2010 IEEE International Conference on Robotics and Automation.

[34]  Jorge Pomares,et al.  Direct Visual Servoing to Track Trajectories in Human-Robot Cooperation , 2011 .

[35]  Greg Chance,et al.  An assistive robot to support dressing - strategies for planning and error handling , 2016, 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[36]  Behraad Bahreyni,et al.  Tri-Mode Capacitive Proximity Detection Towards Improved Safety in Industrial Robotics , 2018, IEEE Sensors Journal.

[37]  Kenneth C. Smith,et al.  A multi-touch three dimensional touch-sensitive tablet , 1985, CHI '85.

[38]  Petros Maragos,et al.  Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot , 2017, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[39]  Euisik Yoon,et al.  Dual-Mode Capacitive Proximity Sensor for Robot Application: Implementation of Tactile and Proximity Sensing Capability on a Single Polymer Platform Using Shared Electrodes , 2009, IEEE Sensors Journal.

[40]  C. Karen Liu,et al.  What does the person feel? Learning to infer applied forces during robot-assisted dressing , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[41]  Heinz Wörn,et al.  6D proximity servoing for preshaping and haptic exploration using capacitive tactile proximity sensors , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[42]  François Chaumette,et al.  Visual servo control. II. Advanced approaches [Tutorial] , 2007, IEEE Robotics & Automation Magazine.

[43]  C. Karen Liu,et al.  Assistive Gym: A Physics Simulation Framework for Assistive Robotics , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[44]  Joseph A. Paradiso,et al.  Applying electric field sensing to human-computer interfaces , 1995, CHI '95.

[45]  Fan Zhang,et al.  Personalized robot-assisted dressing using user modeling in latent spaces , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[46]  Nishanth Koganti,et al.  Bayesian Nonparametric Learning of Cloth Models for Real-Time State Estimation , 2017, IEEE Transactions on Robotics.

[47]  Charles C. Kemp,et al.  3D Human Pose Estimation on a Configurable Bed from a Pressure Image , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[48]  Fan Zhang,et al.  Probabilistic Real-Time User Posture Tracking for Personalized Robot-Assisted Dressing , 2019, IEEE Transactions on Robotics.

[49]  Marco Crepaldi,et al.  A Robust Capacitive Digital Read-Out Circuit for a Scalable Tactile Skin , 2017, IEEE Sensors Journal.

[50]  Björn Hein,et al.  Material Recognition Using a Capacitive Proximity Sensor with Flexible Spatial Resolution , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[51]  Eyal Ofek,et al.  TORC: A Virtual Reality Controller for In-Hand High-Dexterity Finger Interaction , 2019, CHI.

[52]  Nima Najmaei,et al.  Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging – techniques and applications , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[53]  Carme Torras,et al.  Robot-Aided Cloth Classification Using Depth Information and CNNs , 2016, AMDO.

[54]  Patrick Rives,et al.  A new approach to visual servoing in robotics , 1992, IEEE Trans. Robotics Autom..

[55]  Yevgen Chebotar,et al.  Learning Latent Space Dynamics for Tactile Servoing , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[56]  Mark R. Cutkosky,et al.  Capacitive Sensing for a Gripper With Gecko-Inspired Adhesive Film , 2019, IEEE Robotics and Automation Letters.

[57]  Behraad Bahreyni,et al.  Multi-functional capacitive proximity sensing system for industrial safety applications , 2016, 2016 IEEE SENSORS.

[58]  Greg Chance,et al.  A Quantitative Analysis of Dressing Dynamics for Robotic Dressing Assistance , 2017, Front. Robot. AI.

[59]  François Chaumette,et al.  Visual servo control. I. Basic approaches , 2006, IEEE Robotics & Automation Magazine.

[60]  Fan Zhang,et al.  Learning Grasping Points for Garment Manipulation in Robot-Assisted Dressing , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[61]  Vincent Duchaine,et al.  A highly sensitive multimodal capacitive tactile sensor , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[62]  Matthew S. Reynolds,et al.  Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction , 2017, CHI.

[63]  Petros Maragos,et al.  I-Support: A robotic platform of an assistive bathing robot for the elderly population , 2020, Robotics Auton. Syst..

[64]  H. B. Muhammad,et al.  A capacitive tactile sensor array for surface texture discrimination , 2011 .

[65]  Yiannis Demiris,et al.  User modelling for personalised dressing assistance by humanoid robots , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[66]  Charles C. Kemp,et al.  Tracking Human Pose During Robot-Assisted Dressing Using Single-Axis Capacitive Proximity Sensing , 2017, IEEE Robotics and Automation Letters.

[67]  Hubert Zangl,et al.  Responsive fingers — capacitive sensing during object manipulation , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[68]  Wendy A. Rogers,et al.  Identifying the Potential for Robotics to Assist Older Adults in Different Living Environments , 2014, Int. J. Soc. Robotics.

[69]  Nathan F. Lepora,et al.  Exploratory Tactile Servoing With Active Touch , 2017, IEEE Robotics and Automation Letters.

[70]  Heinz Woern,et al.  A tactile proximity sensor , 2010, 2010 IEEE Sensors.

[71]  Long Wang,et al.  Locomotion Mode Classification Using a Wearable Capacitive Sensing System , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[72]  Nassir Navab,et al.  Patient MoCap: Human Pose Estimation Under Blanket Occlusion for Hospital Monitoring Applications , 2016, MICCAI.

[73]  Manuela M. Veloso,et al.  Personalized Assistance for Dressing Users , 2015, ICSR.

[74]  Mark R. Cutkosky,et al.  A Stretchable Capacitive Sensory Skin for Exploring Cluttered Environments , 2020, IEEE Robotics and Automation Letters.

[75]  Wolfgang Reif,et al.  Robust Distance Estimation of Capacitive Proximity Sensors in HRI using Neural Networks , 2020, 2020 Fourth IEEE International Conference on Robotic Computing (IRC).

[76]  Joshua R. Smith,et al.  Electric Field Servoing for robotic manipulation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[77]  Chiya Zhang,et al.  A Review on Applications of Capacitive Displacement Sensing for Capacitive Proximity Sensor , 2020, IEEE Access.

[78]  Paul Lukowicz,et al.  Active Capacitive Sensing: Exploring a New Wearable Sensing Modality for Activity Recognition , 2010, Pervasive.

[79]  Christian Duriez,et al.  A Model-Based Sensor Fusion Approach for Force and Shape Estimation in Soft Robotics , 2020, IEEE Robotics and Automation Letters.

[80]  Stefan Koch,et al.  3D contour following for a cylindrical end-effector using capacitive proximity sensors , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[81]  C. Karen Liu,et al.  Deep Haptic Model Predictive Control for Robot-Assisted Dressing , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[82]  Greg Chance,et al.  “Elbows Out”—Predictive Tracking of Partially Occluded Pose for Robot-Assisted Dressing , 2018, IEEE Robotics and Automation Letters.

[83]  Nishanth Koganti,et al.  A framework for robotic clothing assistance by imitation learning , 2019, Adv. Robotics.

[84]  Enhao Zheng,et al.  Forearm Motion Recognition With Noncontact Capacitive Sensing , 2018, Front. Neurorobot..

[85]  Michael Mende,et al.  A versatile and modular capacitive tactile proximity sensor , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[86]  S. Jeschke,et al.  A Visual Servoing System for Interactive Human-Robot Object Transfer , 2014 .

[87]  D. Cotton,et al.  A Multifunctional Capacitive Sensor for Stretchable Electronic Skins , 2009, IEEE Sensors Journal.

[88]  Giorgio Metta,et al.  Carbon Nanofiber versus Graphene‐Based Stretchable Capacitive Touch Sensors for Artificial Electronic Skin , 2017, Advanced science.

[89]  Mark R. Cutkosky,et al.  A robust, low-cost and low-noise artificial skin for human-friendly robots , 2010, 2010 IEEE International Conference on Robotics and Automation.

[90]  Albrecht Schmidt,et al.  Evaluating capacitive touch input on clothes , 2008, Mobile HCI.

[91]  H. Worn,et al.  Tactile proximity sensors for robotic applications , 2013, 2013 IEEE International Conference on Industrial Technology (ICIT).

[92]  J. Pomares,et al.  Visual servoing path tracking for safe human-robot interaction , 2009, 2009 IEEE International Conference on Mechatronics.

[93]  Enhao Zheng,et al.  Noncontact Capacitive Sensing-Based Locomotion Transition Recognition for Amputees With Robotic Transtibial Prostheses , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[94]  Tao Chen,et al.  The Design and Characterization of a Flexible Tactile Sensing Array for Robot Skin , 2016, Sensors.