Learning Grasp Affordance Reasoning Through Semantic Relations

Reasoning about object affordances allows an autonomous agent to perform generalised manipulation tasks among object instances. While current approaches to grasp affordance estimation are effective, they are limited to a single hypothesis. We present an approach for detection and extraction of multiple grasp affordances on an object via visual input. We define semantics as a combination of multiple attributes, which yields benefits in terms of generalisation for grasp affordance prediction. We use Markov Logic Networks to build a knowledge base graph representation to obtain a probability distribution of grasp affordances for an object. To harvest the knowledge base, we collect and make available a novel dataset that relates different semantic attributes. We achieve reliable mappings of the predicted grasp affordances on the object by learning prototypical grasping patches from several examples. We show our method's generalisation capabilities on grasp affordance prediction for novel instances and compare with similar methods in the literature. Moreover, using a robotic platform, on simulated and real scenarios, we evaluate the success of the grasping task when conditioned on the grasp affordance prediction.

[1]  Ali Farhadi,et al.  Describing objects by their attributes , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Mark Steedman,et al.  Object-Action Complexes: Grounded abstractions of sensory-motor processes , 2011, Robotics Auton. Syst..

[3]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Michael A. Arbib,et al.  Learning to grasp and extract affordances: the Integrated Learning of Grasps and Affordances (ILGA) model , 2015, Biological Cybernetics.

[5]  Manuel Lopes,et al.  Learning grasping affordances from local visual descriptors , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[6]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[7]  Ayellet Tal,et al.  Hierarchical mesh decomposition using fuzzy clustering and cuts , 2003, ACM Trans. Graph..

[8]  三嶋 博之.  The theory of affordances , 2008 .

[9]  Ashutosh Saxena,et al.  Efficient grasping from RGBD images: Learning using a new rectangle representation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[10]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[11]  Nikolaos G. Tsagarakis,et al.  Object-based affordances detection with Convolutional Neural Networks and dense Conditional Random Fields , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[12]  Ali Farhadi,et al.  AI2-THOR: An Interactive 3D Environment for Visual AI , 2017, ArXiv.

[13]  Luc De Raedt,et al.  Learning relational affordance models for robots in multi-object manipulation tasks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[14]  N. Kruger,et al.  Learning object-specific grasp affordance densities , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[15]  Stephen Hart,et al.  The Affordance Template ROS package for robot task programming , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Darwin G. Caldwell,et al.  AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[17]  Markus Vincze,et al.  3DNet: Large-scale object class recognition from CAD models , 2012, 2012 IEEE International Conference on Robotics and Automation.

[18]  Antonio Torralba,et al.  Recognizing indoor scenes , 2009, CVPR.

[19]  Danica Kragic,et al.  Learning task constraints for robot grasping using graphical models , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Subramanian Ramamoorthy,et al.  Towards robust grasps: Using the environment semantics for robotic object affordances , 2018, AAAI 2018.

[21]  Danica Kragic,et al.  Learning grasping points with shape context , 2010, Robotics Auton. Syst..

[22]  Johnson,et al.  Modified Broyden's method for accelerating convergence in self-consistent calculations. , 1988, Physical review. B, Condensed matter.

[23]  G. Rizzolatti,et al.  Visuomotor neurons: ambiguity of the discharge or 'motor' perception? , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[24]  Li Fei-Fei,et al.  Reasoning about Object Affordances in a Knowledge Base Representation , 2014, ECCV.

[25]  Ali Farhadi,et al.  Target-driven visual navigation in indoor scenes using deep reinforcement learning , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Chang‐Jin Kim,et al.  State-Space Models with Regime-Switching: Classical and Gibbs Sampling Approaches with Applications , 1999 .