What Can I Do With This Tool? Self-Supervised Learning of Tool Affordances From Their 3-D Geometry

The ability to use tools can significantly increase the range of activities that an agent is capable of. Humans start using external objects since an early age to accomplish their goals, learning from interaction and observation the relationship between the objects used, their own actions, and the resulting effects, i.e., the tool affordances. Robots capable of autonomously learning affordances in a similar self-supervised way would be far more versatile and simpler to design than purpose-specific ones. This paper proposes and evaluates an approach to allow robots to learn tool affordances from interaction, and generalize them among similar tools based on their 3-D geometry. A set of actions is performed by the iCub robot with a large number of tools grasped in different poses, and the effects observed. Tool affordances are learned as a regression between tool-pose features and action-effect vector projections on respective self-organizing maps, which enables the system to avoid categorization and keep gradual representations of both elements. Moreover, we propose a set of robot-centric 3-D tool descriptors, and study their suitability for interaction scenarios, comparing also their performance against features derived from deep convolutional neural networks. Results show that the presented methods allow the robot to predict the effect of its tool use actions accurately, even for previously unseen tool and poses, and thereby to select the best action for a particular goal given a tool-pose.

[1]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[2]  M. Turvey.  Affordances and Prospective Control: An Outline of the Ontology , 1992 .

[3]  Berthold K. P. Horn.  Extended Gaussian images , 1984, Proceedings of the IEEE.

[4]  Pierre-Yves Oudeyer,et al.  Active learning of inverse models with intrinsically motivated goal exploration in robots , 2013, Robotics Auton. Syst..

[5]  Alessandro Roncone,et al.  3D stereo estimation and fully automated learning of eye-hand coordination in humanoid robots , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[6]  Robert L. Goldstone,et al.  Definition , 1960, A Philosopher Looks at Sport.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Frank Guerin,et al.  A model-based approach to finding substitute tools in 3D vision data , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Alexandre Bernardino,et al.  Learning visual affordances of objects and tools through autonomous robot exploration , 2014, 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[10]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[11]  Justus H. Piater,et al.  Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Nikolaos G. Tsagarakis,et al.  Detecting object affordances with Convolutional Neural Networks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Juergen Gall,et al.  Weakly Supervised Learning of Affordances , 2016, ArXiv.

[15]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[16]  Dejan V. Vranic,et al.  3D model retrieval , 2004 .

[17]  Justus H. Piater,et al.  Emergent structuring of interdependent affordance learning tasks , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[18]  Marc'Aurelio Ranzato,et al.  Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Jivko Sinapov,et al.  A Behavior-Grounded Approach to Forming Object Categories: Separating Containers From Noncontainers , 2012, IEEE Transactions on Autonomous Mental Development.

[20]  Giorgio Metta,et al.  Multi-model approach based on 3D functional features for tool affordance learning in robotics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[21]  Emre Ugur,et al.  Emergent Structuring of Interdependent Affordance Learning Tasks Using Intrinsic Motivation and Empirical Feature Selection , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[22]  Angelo Cangelosi,et al.  An open-source simulator for cognitive robotics research: the prototype of the iCub humanoid robot simulator , 2008, PerMIS.

[23]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  A. Stoytchev.  Toward Learning the Binding Affordances of Objects : A Behavior-Grounded Approach , 2022 .

[25]  E. Gibson,et al.  An Ecological Approach to Perceptual Learning and Development , 2000 .

[26]  Ian D. Reid,et al.  STAR3D: Simultaneous Tracking and Reconstruction of 3D Objects Using RGB-D Data , 2013, 2013 IEEE International Conference on Computer Vision.

[27]  Geoffrey E. Hinton,et al.  3D Object Recognition with Deep Belief Nets , 2009, NIPS.

[28]  Tetsunari Inamura,et al.  Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools , 2013, Artificial Life and Robotics.

[29]  Tetsunari Inamura,et al.  Learning of Tool Affordances for autonomous tool manipulation , 2011, 2011 IEEE/SICE International Symposium on System Integration (SII).

[30]  Justus H. Piater,et al.  Bootstrapping paired-object affordance learning with learned single-affordance features , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[31]  E. Gibson,et al.  Principles of Perceptual Learning and Development , 1973 .

[32]  Atabak Dehban,et al.  Denoising auto-encoders for learning of objects and tools affordances in continuous space , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Esa Alhoniemi,et al.  Publication 6 SelfOrganizing Map in Matlab: the SOM Toolbox , 1999 .

[34]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[35]  Angelo Cangelosi,et al.  Affordances in Psychology, Neuroscience, and Robotics: A Survey , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[36]  Luc De Raedt,et al.  Learning relational affordance models for robots in multi-object manipulation tasks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[37]  Mark Steedman,et al.  Object-Action Complexes: Grounded abstractions of sensory-motor processes , 2011, Robotics Auton. Syst..

[38]  Manuel Lopes,et al.  Modeling affordances using Bayesian networks , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[40]  Yann LeCun,et al.  Learning Invariant Feature Hierarchies , 2012, ECCV Workshops.

[41]  Alexandre Bernardino,et al.  Learning intermediate object affordances: Towards the development of a tool concept , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[42]  Giulio Sandini,et al.  Learning about objects through action - initial steps towards artificial cognition , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[43]  Dirk Kraft,et al.  A Survey of the Ontogeny of Tool Use: From Sensorimotor Experience to Planning , 2013, IEEE Transactions on Autonomous Mental Development.

[44]  Emre Ugur,et al.  Going beyond the perception of affordances: Learning how to actualize them through behavioral parameters , 2011, 2011 IEEE International Conference on Robotics and Automation.

[45]  Danijel Skocaj,et al.  Self-supervised cross-modal online learning of basic object affordances for developmental robotic systems , 2010, 2010 IEEE International Conference on Robotics and Automation.

[46]  Alexander Stoytchev,et al.  Behavior-Grounded Representation of Tool Affordances , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[47]  Jianfei Cai,et al.  Kinect-Based Easy 3D Object Reconstruction , 2012, PCM.

[48]  Danica Kragic,et al.  Improving generalization for 3D object categorization with Global Structure Histograms , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Claude Sammut,et al.  Tool Use Learning in Robots , 2011, AAAI Fall Symposium: Advances in Cognitive Systems.

[50]  Gaurav S. Sukhatme,et al.  Semantic labeling of 3D point clouds with object affordance for robot manipulation , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[51]  M. Viezzer,et al.  Learning affordance concepts : some seminal ideas , 2022 .

[52]  G. Luppino,et al.  Parietofrontal Circuits for Action and Space Perception in the Macaque Monkey , 2001, NeuroImage.

[53]  T. Ziemke,et al.  Theories and computational models of affordance and mirror systems: An integrative review , 2013, Neuroscience & Biobehavioral Reviews.

[54]  Y. Aloimonos,et al.  Affordance of Object Parts from Geometric Features , 2014 .

[55]  G. Rizzolatti,et al.  Object representation in the ventral premotor cortex (area F5) of the monkey. , 1997, Journal of neurophysiology.

[56]  Alexandre Bernardino,et al.  Gaussian mixture models for affordance learning using Bayesian Networks , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[57]  J. Sinapov,et al.  Detecting the functional similarities between tools using a hierarchical representation of outcomes , 2008, 2008 7th IEEE International Conference on Development and Learning.

[58]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Florentin Wörgötter,et al.  Bootstrapping the Semantics of Tools: Affordance Analysis of Real World Objects on a Per-part Basis , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[60]  Lorenzo Rosasco,et al.  Teaching iCub to recognize objects using deep Convolutional Neural Networks , 2015, MLIS@ICML.

[61]  Giulio Sandini,et al.  The iCub humanoid robot: An open-systems platform for research in cognitive development , 2010, Neural Networks.

[62]  Céline Loscos,et al.  3D Model Retrieval , 2013 .

[63]  Ales Ude,et al.  Self-Supervised Online Learning of Basic Object Push Affordances , 2015 .

[64]  Yale Chang,et al.  Unsupervised Feature Learning via Sparse Hierarchical Representations [ 1 ] , 2014 .

[65]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[66]  Sinisa Todorovic,et al.  A Multi-scale CNN for Affordance Segmentation in RGB Images , 2016, ECCV.

[67]  M. Dogar,et al.  Afford or Not to Afford : A New Formalization of Affordances Toward Affordance-Based Robot , 2007 .

[68]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[69]  A. Chemero.  An Outline of a Theory of Affordances , 2003, How Shall Affordances be Refined? Four Perspectives.

[70]  Dirk Kraft,et al.  Learning spatial relationships from 3D vision using histograms , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[71]  Arren Glover,et al.  Developing grounded representations for robots through the principles of sensorimotor coordination , 2014 .

[72]  Giorgio Metta,et al.  Self-supervised learning of grasp dependent tool affordances on the iCub Humanoid robot , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[73]  Alessandro D’Ausilio,et al.  Representing tools as hand movements: Early and somatotopic visuomotor transformations , 2014, Neuropsychologia.

[74]  Christopher W. Geib,et al.  Object Action Complexes as an Interface for Planning and Robot Control , 2006 .

[75]  G. Metta,et al.  Exploring affordances and tool use on the iCub , 2013, 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids).

[76]  Emre Ugur,et al.  Goal emulation and planning in perceptual space using learned affordances , 2011, Robotics Auton. Syst..

[77]  Koen V. Hindriks,et al.  Robot learning and use of affordances in goal-directed tasks , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.