Cross-modal visuo-tactile object recognition using robotic active exploration

In this work, we propose a framework to deal with cross-modal visuo-tactile object recognition. By cross-modal visuo-tactile object recognition, we mean that the object recognition algorithm is trained only with visual data and is able to recognize objects leveraging only tactile perception. The proposed cross-modal framework is constituted by three main elements. The first is a unified representation of visual and tactile data, which is suitable for cross-modal perception. The second is a set of features able to encode the chosen representation for classification applications. The third is a supervised learning algorithm, which takes advantage of the chosen descriptor. In order to show the results of our approach, we performed experiments with 15 objects common in domestic and industrial environments. Moreover, we compare the performance of the proposed framework with the performance of 10 humans in a simple cross-modal recognition task.

[1]  Pietro Falco,et al.  Integrated force/tactile sensing: The enabling technology for slipping detection and avoidance , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[2]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[3]  Kaspar Althoefer,et al.  Surface material recognition through haptic exploration using an intelligent contact sensing finger , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Gregory D. Hager,et al.  Tactile-Object Recognition From Appearance Information , 2011, IEEE Transactions on Robotics.

[5]  Danica Kragic,et al.  Enhancing visual perception of shape through tactile glances , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Nico Blodow,et al.  Aligning point cloud views using persistent feature histograms , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Andrew Y. Ng,et al.  Integrating Visual and Range Data for Robotic Object Detection , 2008, ECCV 2008.

[8]  M. Ani Hsieh,et al.  A triangle histogram for object classification by tactile sensing , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Jeannette Bohg,et al.  Fusing visual and tactile sensing for 3-D object reconstruction while grasping , 2013, 2013 IEEE International Conference on Robotics and Automation.

[10]  Dimitar Dimitrov,et al.  Extraction of grasp-related features by human dual-hand object exploration , 2009, 2009 International Conference on Advanced Robotics.

[11]  Helge J. Ritter,et al.  A Probabilistic Approach to Tactile Shape Reconstruction , 2011, IEEE Transactions on Robotics.

[12]  Gerhard von der Emde,et al.  Cross-modal object recognition and dynamic weighting of sensory inputs in a fish , 2016, Proceedings of the National Academy of Sciences.

[13]  Dongheui Lee,et al.  Fast Visual Odometry Using Intensity-Assisted Iterative Closest Point , 2016, IEEE Robotics and Automation Letters.

[14]  Luigi Villani,et al.  A Conformable Force/Tactile Skin for Physical Human–Robot Interaction , 2016, IEEE Robotics and Automation Letters.

[15]  Boyer D. Winters,et al.  The neural bases of crossmodal object recognition in non-human primates and rodents: A review , 2015, Behavioural Brain Research.

[16]  Markus Vincze,et al.  Ensemble of shape functions for 3D object classification , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[17]  Yang Gao,et al.  Deep learning for tactile understanding from visual and haptic data , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Kaspar Althoefer,et al.  Novel Tactile-SIFT Descriptor for Object Shape Recognition , 2015, IEEE Sensors Journal.

[19]  Wolfram Burgard,et al.  Object identification with tactile sensors using bag-of-features , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  G. De Maria,et al.  An Artificial Skin Based on Optoelectronic Technology , 2014 .

[21]  C. M. Rogers,et al.  Cross modal perception in apes. , 1973, Neuropsychologia.

[22]  D. Kalman.  A Singularly Valuable Decomposition: The SVD of a Matrix , 1996 .

[23]  D. Levin,et al.  Mesh-Independent Surface Interpolation , 2004 .

[24]  Federico Tombari,et al.  SHOT: Unique signatures of histograms for surface and texture description , 2014, Comput. Vis. Image Underst..

[25]  A. Meltzoff,et al.  Intermodal matching by human neonates , 1979, Nature.

[26]  C. Spence,et al.  The Handbook of Multisensory Processing , 2004 .

[27]  Gordon Cheng,et al.  Re-using prior tactile experience by robotic hands to discriminate in-hand objects via texture properties , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[28]  A. Johnson.  A Representation for 3D Surface Matching , 1997 .