Semi-Supervised Haptic Material Recognition for Robots using Generative Adversarial Networks

Material recognition enables robots to incorporate knowledge of material properties into their interactions with everyday objects. For example, material recognition opens up opportunities for clearer communication with a robot, such as "bring me the metal coffee mug", and recognizing plastic versus metal is crucial when using a microwave or oven. However, collecting labeled training data with a robot is often more difficult than unlabeled data. We present a semi-supervised learning approach for material recognition that uses generative adversarial networks (GANs) with haptic features such as force, temperature, and vibration. Our approach achieves state-of-the-art results and enables a robot to estimate the material class of household objects with ~90% accuracy when 92% of the training data are unlabeled. We explore how well this approach can recognize the material of new objects and we discuss challenges facing generalization. To motivate learning from unlabeled training data, we also compare results against several common supervised learning classifiers. In addition, we have released the dataset used for this work which consists of time-series haptic measurements from a robot that conducted thousands of interactions with 72 household objects.

[1]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[2]  T. Martin McGinnity,et al.  Material classification based on thermal properties — A robot and human evaluation , 2013, 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[3]  Jost Tobias Springenberg,et al.  Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks , 2015, ICLR.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Tapani Raiko,et al.  Semi-supervised Learning with Ladder Networks , 2015, NIPS.

[8]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[9]  Maurizio Valle,et al.  Tactile-Data Classification of Contact Materials Using Computational Intelligence , 2011, IEEE Transactions on Robotics.

[10]  Yang Gao,et al.  Deep learning for tactile understanding from visual and haptic data , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Max Welling,et al.  Semi-supervised Learning with Deep Generative Models , 2014, NIPS.

[12]  Kaspar Althoefer,et al.  Tactile Object Recognition with Semi-Supervised Learning , 2015, ICIRA.

[13]  Geoffrey Zweig,et al.  Recent advances in deep learning for speech research at Microsoft , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  Connor Schenck,et al.  Grounding semantic categories in behavioral interactions: Experiments with 100 objects , 2014, Robotics Auton. Syst..

[15]  Roderick Murray-Smith,et al.  Stane: synthesized surfaces for tactile input , 2008, CHI.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[18]  Charles C. Kemp,et al.  Material Recognition from Heat Transfer given Varying Initial Conditions and Short-Duration Contact , 2015, Robotics: Science and Systems.

[19]  Peter J. Ramadge,et al.  Learning to identify container contents through tactile vibration signatures , 2016, 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR).

[20]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[21]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[22]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[23]  Charles C. Kemp,et al.  Data-driven thermal recognition of contact with people and objects , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[24]  Koh Hosoda,et al.  Robust material discrimination by a soft anthropomorphic finger with tactile and thermal sense , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Trevor Darrell,et al.  Using robotic exploratory procedures to learn the meaning of haptic adjectives , 2013, 2013 IEEE International Conference on Robotics and Automation.

[26]  Andrew Owens,et al.  Shape-independent hardness estimation using deep learning and a GelSight tactile sensor , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).