PyTouch: A Machine Learning Library for Touch Processing

With the increased availability of rich tactile sensors, there is an an equally proportional need for open-source and integrated software capable of efficiently and effectively processing raw touch measurements into high-level signals that can be used for control and decision-making. In this paper, we present PyTouch – the first machine learning library dedicated to the processing of touch sensing signals. PyTouch, is designed to be modular, easy-to-use and provides state-of-the-art touch processing capabilities as a service with the goal of unifying the tactile sensing community by providing a library for building scalable, proven, and performance-validated modules over which applications and research can be built upon. We evaluate PyTouch on real-world data from several tactile sensors on touch processing tasks such as touch detection, slip and object pose estimations. PyTouch is open-sourced at https://github.com/facebookresearch/pytouch.

[1]  Edward H. Adelson,et al.  GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force , 2017, Sensors.

[2]  Heather Culbertson,et al.  The Penn Haptic Texture Toolkit for Modeling, Rendering, and Evaluating Haptic Virtual Textures , 2014 .

[3]  Fulvio Mastrogiovanni,et al.  A real-time data acquisition and processing framework for large-scale robot skin , 2015, Robotics Auton. Syst..

[4]  José García Rodríguez,et al.  TactileGCN: A Graph Convolutional Network for Predicting Grasp Stability with Tactile Sensors , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[5]  Alexander C. Abad,et al.  Visuotactile Sensors With Emphasis on GelSight Sensor: A Review , 2020, IEEE Sensors Journal.

[6]  Jan Peters,et al.  Building a Library of Tactile Skills Based on FingerVision , 2019, 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids).

[7]  Jitendra Malik,et al.  More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch , 2018, IEEE Robotics and Automation Letters.

[8]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Fulvio Mastrogiovanni,et al.  Skinware 2.0: A real-time middleware for robot skin , 2015 .

[11]  Jianhua Li,et al.  Slip Detection with Combined Tactile and Visual Information , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Yang Gao,et al.  Proton: A visuo-haptic data acquisition system for robotic learning of surface properties , 2016, 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[13]  Sergey Levine,et al.  OmniTact: A Multi-Directional High-Resolution Touch Sensor , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[15]  Mike Lambeta,et al.  DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation , 2020, IEEE Robotics and Automation Letters.

[16]  Yang Gao,et al.  Deep learning for tactile understanding from visual and haptic data , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).