A brief review of affordance in robotic manipulation research

Abstract This paper presents a brief review of affordance research in robotics, with special concentrations on its applications in grasping and manipulation of objects. The concept of affordance could be a key to realize human-like advanced manipulation intelligence. First, we discuss the concept of affordance while associating with the applications in robotics. Then, we intensively explore the studies that utilize affordance for robotic manipulation applications, such as object recognition, grasping, and object manipulation including tool-use. They obtain and use affordance by several ways like learning from human, using simulation, and real-world execution. Moreover, we show our current work, which is a cloud database for advanced manipulation intelligence. The database accumulates various data related to manipulation task execution and will be an open platform to leverage various affordance techniques. Graphical Abstract

[1]  Edwin Olson,et al.  Predicting object functionality using physical simulations , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Ales Ude,et al.  Comparison of action-grounded and non-action-grounded 3-D shape features for object affordance classification , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[3]  R. Amant,et al.  Affordances for robots: a brief survey , 2012 .

[4]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[5]  T. Stoffregen.  Affordances as Properties of the Animal-Environment System , 2003, How Shall Affordances be Refined? Four Perspectives.

[6]  Stephen Hart,et al.  The Affordance Template ROS package for robot task programming , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Oskar von Stryk,et al.  Template-based manipulation in unstructured environments for supervised semi-autonomous humanoid robots , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[8]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[9]  Danica Kragic,et al.  Generalizing grasps across partly similar objects , 2012, 2012 IEEE International Conference on Robotics and Automation.

[10]  Danica Kragic,et al.  Exploring affordances in robot grasping through latent structure representation , 2010, ECCV 2010.

[11]  Markus Vincze,et al.  AfNet: The Affordance Network , 2012, Asian Conference on Computer Vision.

[12]  Siddhartha S. Srinivasa,et al.  Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set , 2015, IEEE Robotics & Automation Magazine.

[13]  Tamim Asfour,et al.  The KIT whole-body human motion database , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[14]  R. Shaw,et al.  Perceiving, Acting and Knowing : Toward an Ecological Psychology , 1978 .

[15]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[16]  Garth Zeglin,et al.  Preparatory object rotation as a human-inspired grasping strategy , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[17]  Shaogang Ren,et al.  Human-object-object-interaction affordance , 2013, 2013 IEEE Workshop on Robot Vision (WORV).

[18]  Justus H. Piater,et al.  Bootstrapping paired-object affordance learning with learned single-affordance features , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[19]  Michael S. Ryoo,et al.  Learning Social Affordance for Human-Robot Interaction , 2016, IJCAI.

[20]  Maya Cakmak,et al.  To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control , 2007, Adapt. Behav..

[21]  Danica Kragic,et al.  Visual object-action recognition: Inferring object affordances from human demonstration , 2011, Comput. Vis. Image Underst..

[22]  Kensuke Harada,et al.  A framework for systematic accumulation, sharing and reuse of task implementation knowledge , 2016, 2016 IEEE/SICE International Symposium on System Integration (SII).

[23]  Kensuke Harada,et al.  Extracting grasping, contact points and objects motion from assembly demonstration , 2017, 2017 13th IEEE Conference on Automation Science and Engineering (CASE).

[24]  Justus H. Piater,et al.  Refining grasp affordance models by experience , 2010, 2010 IEEE International Conference on Robotics and Automation.

[25]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[26]  Markus Vincze,et al.  Parallel Deep Learning with Suggestive Activation for Object Category Recognition , 2013, ICVS.

[27]  A. Chemero.  An Outline of a Theory of Affordances , 2003, How Shall Affordances be Refined? Four Perspectives.

[28]  P. L. Adams.  THE ORIGINS OF INTELLIGENCE IN CHILDREN , 1976 .

[29]  Giorgio Metta,et al.  Self-supervised learning of grasp dependent tool affordances on the iCub Humanoid robot , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Koen V. Hindriks,et al.  Active learning of affordances for robot use of household objects , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[31]  Karthik Mahesh Varadarajan,et al.  Topological mapping for robot navigation using affordance features , 2015, 2015 6th International Conference on Automation, Robotics and Applications (ICARA).

[32]  Maya Cakmak,et al.  The learning and use of traversability affordance using range images on a mobile robot , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[33]  Jin-Hui Zhu,et al.  Affordance Research in Developmental Robotics: A Survey , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[34]  Manuel Lopes,et al.  Affordance-based imitation learning in robots , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Ronald C. Arkin,et al.  An Behavior-based Robotics , 1998 .

[36]  Angelo Cangelosi,et al.  Affordances in Psychology, Neuroscience, and Robotics: A Survey , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[37]  Peter K. Allen,et al.  Graspit! A versatile simulator for robotic grasping , 2004, IEEE Robotics & Automation Magazine.

[38]  Kazuhiko Kawamura,et al.  Towards a cognitive robot that uses internal rehearsal to learn affordance relations , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Song-Chun Zhu,et al.  Understanding tools: Task-oriented object modeling, learning and recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Danica Kragic,et al.  Learning grasping affordance using probabilistic and ontological approaches , 2009, 2009 International Conference on Advanced Robotics.

[41]  Joachim Hertzberg,et al.  The Role of Functional Affordances in Socializing Robots , 2015, International Journal of Social Robotics.

[42]  Ashutosh Saxena,et al.  Efficient grasping from RGBD images: Learning using a new rectangle representation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[43]  Florentin Wörgötter,et al.  Bootstrapping the Semantics of Tools: Affordance Analysis of Real World Objects on a Per-part Basis , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[44]  James M. Rehg,et al.  Affordance Prediction via Learned Object Attributes , 2011 .

[45]  J. Gibson.  The Senses Considered As Perceptual Systems , 1967 .

[46]  Keith S. Jones,et al.  What Is an Affordance? , 2003, How Shall Affordances be Refined? Four Perspectives.

[47]  Joachim Hertzberg,et al.  Finding Ways to Get the Job Done: An Affordance-Based Approach , 2014, ICAPS.

[48]  Luc De Raedt,et al.  Learning relational affordance models for robots in multi-object manipulation tasks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[49]  N. Kruger,et al.  Learning object-specific grasp affordance densities , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[50]  Yukie Nagai,et al.  Parental scaffolding as a bootstrapping mechanism for learning grasp affordances and imitation skills , 2014, Robotica.

[51]  Eren Erdal Aksoy,et al.  Towards a hierarchy of loco-manipulation affordances , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[52]  Akira Nakamura,et al.  Modeling of everyday objects for semantic grasp , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[53]  Sinan Kalkan,et al.  Learning Social Affordances and Using Them for Planning , 2013, CogSci.

[54]  Hedvig Kjellström,et al.  Functional object descriptors for human activity modeling , 2013, 2013 IEEE International Conference on Robotics and Automation.

[55]  Markus Vincze,et al.  AfRob: The affordance network ontology for robots , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[56]  Giorgio Metta,et al.  Multi-model approach based on 3D functional features for tool affordance learning in robotics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[57]  J. J. Gibson.  The theory of affordances , 1977 .

[58]  Danica Kragic,et al.  Learning a dictionary of prototypical grasp-predicting parts from grasping experience , 2013, 2013 IEEE International Conference on Robotics and Automation.

[59]  Anis Sahbani,et al.  A hybrid approach for grasping 3D objects , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[60]  Tetsunari Inamura,et al.  Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools , 2013, Artificial Life and Robotics.

[61]  Luc De Raedt,et al.  Occluded object search by relational affordances , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[62]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[63]  Hema Swetha Koppula,et al.  Anticipating Human Activities Using Object Affordances for Reactive Robotic Response , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Koen V. Hindriks,et al.  Robot learning and use of affordances in goal-directed tasks , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[65]  Barbara Caputo,et al.  Using Object Affordances to Improve Object Recognition , 2011, IEEE Transactions on Autonomous Mental Development.

[66]  Luc Van Gool,et al.  What makes a chair a chair? , 2011, CVPR 2011.

[67]  Danica Kragic,et al.  Predicting slippage and learning manipulation affordances through Gaussian Process regression , 2013, 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids).

[68]  Aaron M. Dollar,et al.  Classifying human manipulation behavior , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[69]  James M. Rehg,et al.  Learning Visual Object Categories for Robot Affordance Prediction , 2010, Int. J. Robotics Res..

[70]  Koen V. Hindriks,et al.  Effective transfer learning of affordances for household robots , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[71]  Aaron M. Dollar,et al.  The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments , 2015, Int. J. Robotics Res..

[72]  Richard Fikes,et al.  STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving , 1971, IJCAI.

[73]  Andrew H. Fagg,et al.  Grasping Affordances: Learning to Connet vission to Hand Action , 2009 .

[74]  Oskar von Stryk,et al.  Achieving versatile manipulation tasks with unknown objects by supervised humanoid robots based on object templates , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[75]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[76]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[77]  Florentin Wörgötter,et al.  Context Dependent Action Affordances and their Execution using an Ontology of Actions and 3D Geometric Reasoning , 2018, VISIGRAPP.

[78]  Ergun Bicici,et al.  Reasoning About the Functionality of Tools and Physical Artifacts , 2003 .

[79]  Nikolaos G. Tsagarakis,et al.  Detecting object affordances with Convolutional Neural Networks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[80]  Hedvig Kjellström,et al.  Functional Descriptors for Object Affordances , 2015 .

[81]  Frank Guerin,et al.  A model-based approach to finding substitute tools in 3D vision data , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[82]  Nobutaka Shimada,et al.  Construction of Latent Descriptor Space and Inference Model of Hand-Object Interactions , 2017, IEICE Trans. Inf. Syst..

[83]  明彦 小長谷,et al.  遺伝子ネットワークの推定 (データマイニング・ビジネスプロセスとこう繋げ) -- (データマイニングツール(各社,大学を含む)適用実例集!) , 2003 .

[84]  J. Santos-Victor,et al.  Affordances, development and imitation , 2007, 2007 IEEE 6th International Conference on Development and Learning.

[85]  Giorgio Metta,et al.  Self-supervised learning of tool affordances from 3D tool representation through parallel SOM mapping , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[86]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[87]  Rada Mihalcea,et al.  Mining semantic affordances of visual object categories , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[88]  Mark Steedman,et al.  Plans, Affordances, And Combinatory Grammar , 2002 .

[89]  M. Turvey.  Affordances and Prospective Control: An Outline of the Ontology , 1992 .

[90]  Yiannis Aloimonos,et al.  Affordance detection of tool parts from geometric features , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[91]  Máximo A. Roa,et al.  Functional power grasps transferred through warping and replanning , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[92]  Alexandre Bernardino,et al.  Learning visual affordances of objects and tools through autonomous robot exploration , 2014, 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[93]  Alexei A. Efros,et al.  From 3D scene geometry to human workspace , 2011, CVPR 2011.

[94]  Bernt Schiele,et al.  Functional Object Class Detection Based on Learned Affordance Cues , 2008, ICVS.

[95]  Kohtaro Ohba,et al.  Learning affordance for semantic robots using ontology approach , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.