Proximity Perception in Human-Centered Robotics: A Survey on Sensing Systems and Applications

Proximity perception is a technology that has the potential to play an essential role in the future of robotics. It can fulfill the promise of safe, robust, and autonomous systems in industry and everyday life, alongside humans, as well as in remote locations in space and underwater. In this survey paper, we cover the developments of this field from the early days up to the present, with a focus on human-centered robotics. Here, proximity sensors are typically deployed in two scenarios: first, on the exterior of manipulator arms to support safety and interaction functionality, and second, on the inside of grippers or hands to support grasping and exploration. Starting from this observation, we propose a categorization for the approaches found in the literature. To provide a basis for understanding these approaches, we devote effort to present the technologies and different measuring principles that were developed over the years, also providing a summary in form of a table. Then, we show the diversity of applications that have been presented in the literature. Finally, we give an overview of the most important trends that will shape the future of this domain.

[1]  Hubert Zangl,et al.  Object detection based on electrical capacitance tomography , 2015, 2015 IEEE Sensors Applications Symposium (SAS).

[2]  Jan Steckel,et al.  Acoustic Flow-Based Control of a Mobile Platform Using a 3D Sonar Sensor , 2017, IEEE Sensors Journal.

[3]  Tanir Ozcelebi,et al.  Millimeter Wave Sensing: A Review of Application Pipelines and Building Blocks , 2020, IEEE Sensors Journal.

[4]  Hubert Zangl,et al.  Design of a Novel Gripper System with 3D- and Inkjet-printed Multimodal Sensors for Automated Grasping of a Forestry Robot , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[5]  Michael Mende,et al.  A versatile and modular capacitive tactile proximity sensor , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[6]  Habib Ammari,et al.  Robust Electromagnetic Pose Estimation for Robotic Applications , 2020, IEEE Transactions on Instrumentation and Measurement.

[7]  E. Hall,et al.  The Hidden Dimension , 1970 .

[8]  Andrea Maria Zanchettin,et al.  Safety Control of Industrial Robots Based on a Distributed Distance Sensor , 2014, IEEE Transactions on Control Systems Technology.

[9]  Zhiguang Xiao,et al.  Gripper self-alignment for autonomous pole-grasping with a biped climbing robot , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[10]  Masatoshi Ishikawa,et al.  High-Speed, Small-Deformation Catching of Soft Objects Based on Active Vision and Proximity Sensing , 2019, IEEE Robotics and Automation Letters.

[11]  Claude Samson,et al.  Application of the Task-function Approach to Sensor-based Control of Robot Manipulators , 1990 .

[12]  A. R. Johnson Optical proximity sensors for manipulators , 1973 .

[13]  Gordon Cheng,et al.  A Tactile-Based Framework for Active Object Learning and Discrimination using Multimodal Robotic Skin , 2017, IEEE Robotics and Automation Letters.

[14]  M. Buehler,et al.  Dynamic object localization via a proximity sensor network , 1996, 1996 IEEE/SICE/RSJ International Conference on Multisensor Fusion and Integration for Intelligent Systems (Cat. No.96TH8242).

[15]  Cynthia Breazeal,et al.  A Sensitive Skin for Robotic Companions Featuring Temperature, Force, and Electric Field Sensors , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Vladimir J. Lumelsky,et al.  Real-time collision avoidance in teleoperated whole-sensitive robot arm manipulators , 1993, IEEE Trans. Syst. Man Cybern..

[17]  H. Worn,et al.  Tactile proximity sensors for robotic applications , 2013, 2013 IEEE International Conference on Industrial Technology (ICIT).

[18]  Carlos Balaguer,et al.  Adaptive Aid on Targeted Robot Manipulator Movements in Tele-Assistance , 2016, Paladyn J. Behav. Robotics.

[19]  Torsten Kröger,et al.  On-Line Trajectory Generation in Robotic Systems - Basic Concepts for Instantaneous Reactions to Unforeseen (Sensor) Events , 2010, Springer Tracts in Advanced Robotics.

[20]  Hung Van Hoang,et al.  Highly sensitive flexible proximity tactile array sensor by using carbon micro coils , 2017 .

[21]  Aiguo Ming,et al.  Integrated control of a multiple-degree-of-freedom hand and arm using a reactive architecture based on high-speed proximity sensing , 2019, Int. J. Robotics Res..

[22]  Björn Hein,et al.  Material Recognition Using a Capacitive Proximity Sensor with Flexible Spatial Resolution , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Ashutosh Saxena,et al.  Reactive grasping using optical proximity sensors , 2009, 2009 IEEE International Conference on Robotics and Automation.

[24]  Stefan Escaida Navarro,et al.  Teleoperation with Tactile Feedback based on a Capacitive Proximity Sensor Array , 2020, 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[25]  Vladimir J. Lumelsky,et al.  A sensitive skin system for motion control of robot arm manipulators , 1992, Robotics Auton. Syst..

[26]  Carmel Majidi,et al.  Liquid Metal-Microelectronics Integration for a Sensorized Soft Robot Skin , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[27]  Nathan F. Lepora,et al.  TacWhiskers: Biomimetic Optical Tactile Whiskered Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Seok Chang Ryu,et al.  Electrically-Evoked Proximity Sensation Can Enhance Fine Finger Control in Telerobotic Pinch , 2020, Scientific Reports.

[29]  C. Karen Liu,et al.  Multidimensional Capacitive Sensing for Robot-Assisted Dressing and Bathing , 2019, 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR).

[30]  Di Guo,et al.  Transmissive optical pretouch sensing for robotic grasping , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[31]  Antal K. Bejczy,et al.  Planning Considerations for a Roving Robot with Arm , 1973, IJCAI.

[32]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[33]  Michael Mende,et al.  Flexible Spatial Resolution for Preshaping with a Modular Capacitive Tactile Proximity Sensor , 2016, IROS 2016.

[34]  Euisik Yoon,et al.  Dual-Mode Capacitive Proximity Sensor for Robot Application: Implementation of Tactile and Proximity Sensing Capability on a Single Polymer Platform Using Shared Electrodes , 2009, IEEE Sensors Journal.

[35]  Boby George,et al.  A Combined Inductive–Capacitive Proximity Sensor for Seat Occupancy Detection , 2010, IEEE Transactions on Instrumentation and Measurement.

[36]  Michael A. Peshkin,et al.  Finding and identifying simple objects underwater with active electrosense , 2015, Int. J. Robotics Res..

[37]  H. Zangl,et al.  Combined Capacitive and Ultrasonic Distance Measurement for Automotive Applications , 2011, IEEE Sensors Journal.

[38]  Yoji Yamada,et al.  A Proximity-Tactile Sensor to Detect Obstacles for a Cylindrical Arm , 1988, J. Robotics Mechatronics.

[39]  Minas V. Liarokapis,et al.  Single-Grasp, Model-Free Object Classification using a Hyper-Adaptive Hand, Google Soli, and Tactile Sensors , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[40]  Chad DeChant,et al.  Shape completion enabled robotic grasping , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  Gabor J. Tamasy,et al.  Smart sensor networks for robotic sensor skins , 1997 .

[42]  John T. Feddema,et al.  A capacitance-based proximity sensor for whole arm obstacle avoidance , 1992, Proceedings 1992 IEEE International Conference on Robotics and Automation.

[43]  John T. Feddema,et al.  Collision avoidance during teleoperation using whole-arm proximity sensors coupled to a virtual environment , 1993, Other Conferences.

[44]  Carmel Majidi,et al.  Closing the Loop with Liquid-Metal Sensing Skin for Autonomous Soft Robot Gripping , 2020, 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft).

[45]  Vincent Lebastard,et al.  Model based object localization and shape estimation using electric sense on underwater robots , 2017 .

[46]  Nikolaus Correll,et al.  A Robotic Skin for Collision Avoidance and Affective Touch Recognition , 2018, IEEE Robotics and Automation Letters.

[47]  Kevin Huang,et al.  Visionless Tele-Exploration of 3D Moving Objects , 2018, 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[48]  Oussama Khatib,et al.  Virtual whiskers — Highly responsive robot collision avoidance , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Satoshi Tsuji,et al.  Proximity and Contact Sensor for Human Cooperative Robot by Combining Time-of-Flight and Self-Capacitance Sensors , 2020, IEEE Sensors Journal.

[50]  Kris Luyten,et al.  Silicone Devices: A Scalable DIY Approach for Fabricating Self-Contained Multi-Layered Soft Circuits using Microfluidics , 2018, CHI.

[51]  Joshua R. Smith,et al.  Seashell effect pretouch sensing for robotic grasping , 2012, 2012 IEEE International Conference on Robotics and Automation.

[52]  Jesús Ureña,et al.  Review of UAV positioning in indoor environments and new proposal based on US measurements , 2019, IPIN.

[53]  Gordon Cheng,et al.  A Comprehensive Realization of Robot Skin: Sensors, Sensing, Control, and Applications , 2019, Proceedings of the IEEE.

[54]  Hubert Zangl,et al.  Gesture-based Contactless Control of Mobile Manipulators using Capacitive Sensing , 2020, 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[55]  Zhangjing Wang,et al.  Multi-Sensor Fusion in Automated Driving: A Survey , 2020, IEEE Access.

[56]  Joshua R. Smith,et al.  An Electric Field Pretouch system for grasping and co-manipulation , 2010, 2010 IEEE International Conference on Robotics and Automation.

[57]  Joshua R. Smith,et al.  Improved object pose estimation via deep pre-touch sensing , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[58]  John T. Feddema,et al.  Whole arm obstacle avoidance for teleoperated robots , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[59]  Heinz Wörn,et al.  Methods for safe human-robot-interaction using capacitive tactile proximity sensors , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[60]  Wolfgang Reif,et al.  Robust Distance Estimation of Capacitive Proximity Sensors in HRI using Neural Networks , 2020, 2020 Fourth IEEE International Conference on Robotic Computing (IRC).

[61]  Kaspar Althoefer,et al.  Automatic Fracture Characterization Using Tactile and Proximity Optical Sensing , 2020, Frontiers in Robotics and AI.

[62]  Joshua R. Smith,et al.  Electric Field Servoing for robotic manipulation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[63]  M. Buehler,et al.  Robust Estimation of Pre-Contact Object Trajectories , 1997 .

[64]  Johannes Schöning,et al.  Audio-tactile proximity feedback for enhancing 3D manipulation , 2018, VRST.

[65]  Ulrike Thomas,et al.  Collision Avoidance with Proximity Servoing for Redundant Serial Robot Manipulators , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[66]  A. A. Maciejewski,et al.  Obstacle Avoidance , 2005 .

[67]  Aiguo Ming,et al.  Grasping control based on time-to-contact method for a robot hand equipped with proximity sensors on fingertips , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[68]  Gordon Cheng,et al.  Active Tactile Transfer Learning for Object Discrimination in an Unstructured Environment Using Multimodal Robotic Skin , 2017, Int. J. Humanoid Robotics.

[69]  Masatoshi Ishikawa,et al.  High-Speed High-Precision Proximity Sensor for Detection of Tilt, Distance, and Contact , 2018, IEEE Robotics and Automation Letters.

[70]  Heinz Wörn,et al.  6D proximity servoing for preshaping and haptic exploration using capacitive tactile proximity sensors , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[71]  Gamini Dissanayake,et al.  Capacitive sensor for object ranging and material type identification , 2008 .

[72]  Aiguo Ming,et al.  Pre-shaping for various objects by the robot hand equipped with resistor network structure proximity sensors , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[73]  R. A. Russell,et al.  Object location and recognition using whisker sensors , 2003 .

[74]  Oussama Khatib,et al.  Real-Time Obstacle Avoidance for Manipulators and Mobile Robots , 1985, Autonomous Robot Vehicles.

[75]  Jan Steckel,et al.  High-Resolution Ultrasound Sensing for Robotics Using Dense Microphone Arrays , 2020, IEEE Access.

[76]  Siddhartha S. Srinivasa,et al.  Improved Proximity, Contact, and Force Sensing via Optimization of Elastomer-Air Interface Geometry , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[77]  Hubert Zangl,et al.  Electrical Capacitance Tomography with a variable topology , 2015, 2015 9th International Conference on Sensing Technology (ICST).

[78]  Frédéric Boyer,et al.  Model for a Sensor Inspired by Electric Fish , 2012, IEEE Transactions on Robotics.

[79]  Tanja Schultz,et al.  Telemanipulation with force-based display of proximity fields , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[80]  Hubert Zangl,et al.  Contactless Control of a Kinematically Redundant Serial Manipulator Using Tomographic Sensors , 2017, IEEE Robotics and Automation Letters.

[81]  R L Klatzky,et al.  Timecourse of preshaping for functional responses to objects. , 1989, Journal of motor behavior.

[82]  Ulrike Thomas,et al.  With Proximity Servoing towards Safe Human-Robot-Interaction , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[83]  Christian Duriez,et al.  A Model-Based Sensor Fusion Approach for Force and Shape Estimation in Soft Robotics , 2020, IEEE Robotics and Automation Letters.

[84]  John M. Vranish,et al.  Capaciflector collision avoidance sensors for robots , 1991 .

[85]  Stefan Koch,et al.  3D contour following for a cylindrical end-effector using capacitive proximity sensors , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[86]  J. De Schutter,et al.  Improving force controlled planar contour following using online eye-in-hand vision based feedforward , 1999, 1999 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (Cat. No.99TH8399).

[87]  Gordon Cheng,et al.  Humanoid Multimodal Tactile-Sensing Modules , 2011, IEEE Transactions on Robotics.

[88]  Dezhen Song,et al.  Toward Fingertip Non-Contact Material Recognition and Near-Distance Ranging for Robotic Grasping , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[89]  Ulrike Thomas,et al.  Capacitive Proximity Sensor Skin for Contactless Material Detection , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[90]  Gyorgy Cserey,et al.  3D geometry reconstruction using Large Infrared Proximity Array for robotic applications , 2009, 2009 IEEE International Conference on Mechatronics.

[91]  Dimitrios Tzovaras,et al.  3D Underground Mapping with a Mobile Robot and a GPR Antenna , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[92]  Vladimir J. Lumelsky,et al.  Development of sensitive skin for a 3D robot arm operating in an uncertain environment , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[93]  Beno Benhabib,et al.  A novel electrooptical proximity sensor for robotics: calibration and active sensing , 1997, IEEE Trans. Robotics Autom..

[94]  D. Wegcrif,et al.  WHOLE ARM PROXIMITY CONTROL SYSTEM FOR ARTICULATED ROBOTS WORKING NEAR SPACE VEHICLES AND FLIGHT HARDWARE , 1993 .

[95]  Joshua R. Smith,et al.  A unified framework for grasping and shape acquisition via pretouch sensing , 2013, 2013 IEEE International Conference on Robotics and Automation.

[96]  Satoshi Tsuji,et al.  Proximity Skin Sensor Using Time-of-Flight Sensor for Human Collaborative Robot , 2019, IEEE Sensors Journal.

[97]  Daniel G. Wegerif,et al.  Sensor-based whole-arm obstacle avoidance for kinematically redundant robots , 1992, Other Conferences.

[98]  Yosuke Suzuki,et al.  Contact transition control by adjusting emitting energy of proximity sensor , 2021, Adv. Robotics.

[99]  Kaspar Althoefer,et al.  Fingertip proximity sensor with realtime visual-based calibration , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[100]  Marek Teichmann,et al.  Reactive Robotics I: Reactive Grasping with a Modified Gripper and Multifingered Hands , 2000, Int. J. Robotics Res..

[101]  C. Majidi,et al.  Wireless Electronic Skin with Integrated Pressure and Optical Proximity Sensing , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[102]  Matthew S. Reynolds,et al.  2D and 3D millimeter-wave synthetic aperture radar imaging on a PR2 platform , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[103]  Vladimir J. Lumelsky,et al.  Motion planning for robot arm manipulators with proximity sensing , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[104]  Yang Bai,et al.  Biomimetic and bio-inspired robotics in electric fish research , 2013, Journal of Experimental Biology.

[105]  Tien Dat Nguyen,et al.  Skin-Type Proximity Sensor by Using the Change of Electromagnetic Field , 2021, IEEE Transactions on Industrial Electronics.

[106]  Anibal T. de Almeida,et al.  Sensor-based 3-D autonomous contour-following control , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[107]  Kun Jiang,et al.  MMW Radar-Based Technologies in Autonomous Driving: A Review , 2020, Sensors.

[108]  Philippe Fraisse,et al.  Obstacle Avoidance using a Capacitive Skin for Safe Human-Robot Interaction. , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[109]  Aiguo Ming,et al.  Integrated control of a multi-fingered hand and arm using proximity sensors on the fingertips , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[110]  Hubert Zangl,et al.  Radar Sensors in Collaborative Robotics: Fast Simulation and Experimental Validation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[111]  John Kenneth Salisbury,et al.  An Optical Fiber Proximity Sensor for Haptic Exploration , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[112]  Christian Waldschmidt,et al.  160-GHz Radar Proximity Sensor With Distributed and Flexible Antennas for Collaborative Robots , 2019, IEEE Access.

[113]  Massimo Bergamasco,et al.  A fingertip sensor with proximity, tactile and force sensing capabilities , 1990, EEE International Workshop on Intelligent Robots and Systems, Towards a New Frontier of Applications.

[114]  Ivan Poupyrev,et al.  Soli , 2016, ACM Trans. Graph..

[115]  Sarah Bergbreiter,et al.  Effect of finger geometries on strain response of interdigitated capacitor based soft strain sensors , 2018 .

[116]  Joshua R. Smith,et al.  Electric field imaging pretouch for robotic graspers , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[117]  Takeo Kanade,et al.  Design of an Optical Proximity Sensor Using Multiple Cones of Light for Measuring Surface Shape. , 1984 .

[118]  Matthew S. Reynolds,et al.  Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction , 2017, CHI.

[119]  Zhong Lin Wang,et al.  Dual functional transparent film for proximity and pressure sensing , 2014, Nano Research.

[120]  Kaspar Althoefer,et al.  Force and proximity fingertip sensor to enhance grasping perception , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[121]  Mario Huemer,et al.  On the Safe Road Toward Autonomous Driving: Phase noise monitoring in radar sensors for functional safety compliance , 2019, IEEE Signal Processing Magazine.

[122]  Christine Chevallereau,et al.  Underwater Reflex Navigation in Confined Environment Based on Electric Sense , 2013, IEEE Transactions on Robotics.

[123]  Oliver Brock,et al.  Interactive Perception: Leveraging Action in Perception and Perception in Action , 2016, IEEE Transactions on Robotics.

[124]  Charles C. Kemp,et al.  Tracking Human Pose During Robot-Assisted Dressing Using Single-Axis Capacitive Proximity Sensing , 2017, IEEE Robotics and Automation Letters.

[125]  Joshua R. Smith,et al.  Pre-touch sensing for sequential manipulation , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[126]  Hubert Zangl,et al.  Responsive fingers — capacitive sensing during object manipulation , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[127]  Michael Beetz,et al.  Improving robot manipulation through fingertip perception , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[128]  Heinz Woern,et al.  A tactile proximity sensor , 2010, 2010 IEEE Sensors.

[129]  Andrea Maria Zanchettin,et al.  Optimal placement of spots in distributed proximity sensors for safe human-robot interaction , 2013, 2013 IEEE International Conference on Robotics and Automation.

[130]  Nikolaus Correll,et al.  Integrated proximity, contact and force sensing using elastomer-embedded commodity proximity sensors , 2018, Auton. Robots.

[131]  Tien Dat Nguyen,et al.  A highly sensitive dual mode tactile and proximity sensor using Carbon Microcoils for robotic applications , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[132]  Aiguo Ming,et al.  Robotic Grasping Using Proximity Sensors for Detecting both Target Object and Support Surface , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[133]  Carlos Balaguer,et al.  Adaptive collision-limitation behavior for an assistive manipulator , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.