Bio-Inspired Proprioceptive Touch of a Soft Finger with Inner-Finger Kinesthetic Perception

In-hand object pose estimation is challenging for humans and robots due to occlusion caused by the hand and object. This paper proposes a soft finger that integrates inner vision with kinesthetic sensing to estimate object pose inspired by human fingers. The soft finger has a flexible skeleton and skin that adapts to different objects, and the skeleton deformations during interaction provide contact information obtained by the image from the inner camera. The proposed framework is an end-to-end method that uses raw images from soft fingers to estimate in-hand object pose. It consists of an encoder for kinesthetic information processing and an object pose and category estimator. The framework was tested on seven objects, achieving an impressive error of 2.02 mm and 11.34 degrees for pose error and 99.05% for classification.

[1]  Jianwei Zhang,et al.  PoseFusion: Robust Object-in-Hand Pose Estimation with SelectLSTM , 2023, 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Michael F. Wehner,et al.  Proprioceptive Touch of a Soft Actuator Containing an Embedded Intrinsically Soft Sensor using Kinesthetic Feedback , 2023, Journal of Intelligent & Robotic Systems.

[3]  Cewu Lu,et al.  Unseen Object 6D Pose Estimation: A Benchmark and Baselines , 2022, ArXiv.

[4]  Yuan Zhang,et al.  MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  V. Lepetit,et al.  Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation , 2021, Computer Vision and Pattern Recognition.

[6]  Wenzhen Yuan,et al.  Soft magnetic skin for super-resolution tactile sensing with force self-decoupling , 2021, Science Robotics.

[7]  Federico Tombari,et al.  GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jia Pan,et al.  Learning-Based Optoelectronically Innervated Tactile Finger for Rigid-Soft Interactive Grasping , 2021, IEEE Robotics and Automation Letters.

[9]  Maria Bauza,et al.  Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering , 2020, CoRL.

[10]  Haokun Wang,et al.  DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation* , 2020, 2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[11]  Takamitsu Matsubara,et al.  Contact-based in-hand pose estimation using Bayesian state estimation and particle filtering , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[12]  David J. Crandall,et al.  HOPE-Net: A Graph-Based Model for Hand-Object Pose Estimation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Kostas E. Bekris,et al.  Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Mike Lambeta,et al.  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation , 2020, IEEE Robotics and Automation Letters.

[15]  Thomas Brox,et al.  FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Luc Van Gool,et al.  Self-Supervised 3D Hand Pose Estimation Through Training by Fitting , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Michael J. Black,et al.  Learning Joint Reconstruction of Hands and Manipulated Objects , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Marc Pollefeys,et al.  H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Alin Albu-Schäffer,et al.  Fusing Joint Measurements and Visual Features for In-Hand Object Pose Estimation , 2018, IEEE Robotics and Automation Letters.

[20]  Torsten Sattler,et al.  Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Edward H. Adelson,et al.  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force , 2017, Sensors.

[22]  Luis Moreno,et al.  Tactile-Based In-Hand Object Pose Estimation , 2017, ROBOT.

[23]  Christopher G. Atkeson,et al.  Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Christian Szegedy,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[26]  Jens Reinecke,et al.  Online in-hand object localization , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Giulio Sandini,et al.  Tactile Sensing—From Humans to Humanoids , 2010, IEEE Transactions on Robotics.

[28]  R L Klatzky,et al.  Identifying objects by touch: An “expert system” , 1985, Perception & psychophysics.

[29]  Yuan Gao,et al.  In-Hand Pose Estimation Using Hand-Mounted RGB Cameras and Visuotactile Sensors , 2023, IEEE Access.

[30]  Nawid Jamali,et al.  VisuoTactile 6D Pose Estimation of an In-Hand Object using Vision and Tactile Sensor Data , 2022, IEEE Robotics and Automation Letters.

[31]  Fang Wan,et al.  Visual Learning Towards Soft Robot Force Control using a 3D Metamaterial with Differential Stiffness , 2021, CoRL.

[32]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2022 .

[33]  M. T. Mason,et al.  Toward Robotic Manipulation , 2018, Annu. Rev. Control. Robotics Auton. Syst..