Visual Pressure Estimation and Control for Soft Robotic Grippers

—Soft robotic grippers facilitate contact-rich ma- nipulation, including robust grasping of varied objects. Yet the beneficial compliance of a soft gripper also results in significant deformation that can make precision manipulation challenging. We present visual pressure estimation & control (VPEC), a method that uses a single RGB image of an unmodified soft gripper from an external camera to directly infer pressure applied to the world by the gripper. We present inference results for a pneumatic gripper and a tendon-actuated gripper making contact with a flat surface. We also show that VPEC enables precision manipulation via closed-loop control of inferred pressure. We present results for a mobile manipulator (Stretch RE1 from Hello Robot) using visual servoing to do the following: achieve target pressures when making contact; follow a spatial pressure trajectory; and grasp small objects, including a microSD card, a washer, a penny, and a pill. Overall, our results show that VPEC enables grippers with high compliance to perform precision manipulation. 1

[1]  Christopher D. Twigg,et al.  PressureVision: Estimating Hand Pressure from a Single RGB Image , 2022, ECCV.

[2]  The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments , 2021, ArXiv.

[3]  Dukchan Yoon,et al.  Analysis of Fingertip Force Vector for Pinch-Lifting Gripper With Robust Adaptation to Environments , 2021, IEEE Transactions on Robotics.

[4]  Kostas E. Bekris,et al.  Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks , 2021, Robotics: Science and Systems.

[5]  Nathan F. Lepora,et al.  Soft Biomimetic Optical Tactile Sensing With the TacTip: A Review , 2021, IEEE Sensors Journal.

[6]  F. Janabi-Sharifi,et al.  Image-Based Force Estimation in Medical Applications: A Review , 2021, IEEE Sensors Journal.

[7]  N. Lepora,et al.  Pose-Based Tactile Servoing: Controlled Soft Touch Using Deep Learning , 2020, IEEE Robotics & Automation Magazine.

[8]  Oliver Kroemer,et al.  Contact Localization for Robot Arms in Motion without Torque Sensing , 2020, ArXiv.

[9]  A. Jarc,et al.  Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Fernando Torres,et al.  Generation of Tactile Data From 3D Vision and Target Robotic Grasps , 2020, IEEE Transactions on Haptics.

[11]  Jun Kinugawa,et al.  Tactile Servoing Based Pressure Distribution Control of a Manipulator Using a Convolutional Neural Network , 2021, IEEE Access.

[12]  Contact Localization using Velocity Constraints , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Nawid Jamali,et al.  Deep Tactile Experience: Estimating Tactile Sensor Output from Depth Sensor Data , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  Monica Malvezzi,et al.  Hand closure model for planning top grasps with soft robotic hands , 2020, Int. J. Robotics Res..

[15]  Russ Tedrake,et al.  Soft-bubble grippers for robust and perceptive manipulation , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  Michael C. Yip,et al.  Vibration-Based Multi-Axis Force Sensing: Design, Characterization, and Modeling , 2020, IEEE Robotics and Automation Letters.

[17]  Sandra Q. Liu,et al.  Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Wonjun Hwang,et al.  An Efficient Three-Dimensional Convolutional Neural Network for Inferring Physical Interaction Force from Video , 2019, Sensors.

[20]  Antonio Torralba,et al.  Connecting Touch and Vision via Cross-Modal Prediction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Wojciech Samek,et al.  A Recurrent Convolutional Neural Network Approach for Sensorless Force Estimation in Robotic Surgery , 2018, Biomed. Signal Process. Control..

[22]  Aaron M. Dollar,et al.  Robust Precision Manipulation With Simple Process Models Using Visual Servoing Techniques With Disturbance Rejection , 2019, IEEE Transactions on Automation Science and Engineering.

[23]  Vincent Babin,et al.  Picking, grasping, or scooping small objects lying on flat surfaces: A design approach , 2018, Int. J. Robotics Res..

[24]  Mahyar Abdeetedal,et al.  Grasp and Stress Analysis of an Underactuated Finger for Proprioceptive Tactile Sensing , 2018, IEEE/ASME Transactions on Mechatronics.

[25]  Cosimo Della Santina,et al.  Estimating contact forces from postural measures in a class of under-actuated robotic hands , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Oliver Brock,et al.  Visual detection of opportunities to exploit contact in grasping using contextual multi-armed bandits , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[27]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Christopher G. Atkeson,et al.  Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[30]  Lionel Birglen,et al.  Stiffness Analysis of Underactuated Fingers and Its Application to Proprioceptive Tactile Sensing , 2016, IEEE/ASME Transactions on Mechatronics.

[31]  Sergey Levine,et al.  Learning dexterous manipulation for a soft robotic hand from human demonstrations , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[32]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[34]  Edward H. Adelson,et al.  Localization and manipulation of small parts using GelSight tactile sensing , 2014, IROS.

[35]  Milos Zefran,et al.  Using monocular images to estimate interaction forces during minimally invasive surgery , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  J. Andrew Bagnell,et al.  Human-inspired force compliant grasping primitives , 2014, Auton. Robots.

[37]  Francisco José Madrid-Cuevas,et al.  Automatic generation and detection of highly reliable fiducial markers under occlusion , 2014, Pattern Recognit..

[38]  Matei T. Ciocarlie,et al.  The Velo gripper: A versatile single-actuator design for enveloping, parallel and fingertip grasps , 2014, Int. J. Robotics Res..

[39]  Oliver Brock,et al.  Exploitation of environmental constraints in human and robotic grasping , 2015, Int. J. Robotics Res..

[40]  Helge J. Ritter,et al.  A Control Framework for Tactile Servoing , 2013, Robotics: Science and Systems.

[41]  Garth Zeglin,et al.  Measuring contact points from displacements with a compliant, articulated robot hand , 2011, 2011 IEEE International Conference on Robotics and Automation.

[42]  Filip Ilievski,et al.  Soft robotics for chemists. , 2011, Angewandte Chemie.

[43]  Jeremy A. Fishel,et al.  Signal processing and fabrication of a biomimetic tactile sensor array with thermal, force and microvibration modalities , 2009, 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[44]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Cagdas D. Onal,et al.  Visual Servoing-Based Autonomous 2-D Manipulation of Microparticles Using a Nanoprobe , 2007, IEEE Transactions on Control Systems Technology.

[46]  Jaydev P. Desai,et al.  A vision-based approach for estimating contact forces: Applications to robot-assisted surgery , 2005 .

[47]  Eduardo Torres-Jara,et al.  The power of the dark side: using cast shadows for visually-guided touching , 2004, 4th IEEE/RAS International Conference on Humanoid Robots, 2004..

[48]  Bradley J. Nelson,et al.  Modeling elastic objects with neural networks for vision-based force measurement , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[49]  Olac Fuentes,et al.  Experimental evaluation of uncalibrated visual servoing for precision manipulation , 1997, Proceedings of International Conference on Robotics and Automation.

[50]  Peter I. Corke,et al.  A tutorial on visual servo control , 1996, IEEE Trans. Robotics Autom..

[51]  Ning Chen,et al.  Edge tracking using tactile servo , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[52]  Steve Sutphen,et al.  Tactile Servo: Control of Touch-Driven Robot Motion , 1993, ISER.

[53]  Stefan Begej,et al.  Planar and finger-shaped optical tactile sensors for robotic applications , 1988, IEEE J. Robotics Autom..