A brief review of affordance in robotic manipulation research

Abstract This paper presents a brief review of affordance research in robotics, with special concentrations on its applications in grasping and manipulation of objects. The concept of affordance could be a key to realize human-like advanced manipulation intelligence. First, we discuss the concept of affordance while associating with the applications in robotics. Then, we intensively explore the studies that utilize affordance for robotic manipulation applications, such as object recognition, grasping, and object manipulation including tool-use. They obtain and use affordance by several ways like learning from human, using simulation, and real-world execution. Moreover, we show our current work, which is a cloud database for advanced manipulation intelligence. The database accumulates various data related to manipulation task execution and will be an open platform to leverage various affordance techniques. Graphical Abstract

[1]  Florentin Wörgötter,et al.  Context Dependent Action Affordances and their Execution using an Ontology of Actions and 3D Geometric Reasoning , 2018, VISIGRAPP.

[2]  Angelo Cangelosi,et al.  Affordances in Psychology, Neuroscience, and Robotics: A Survey , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[3]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[4]  Keith S. Jones,et al.  What Is an Affordance? , 2003, How Shall Affordances be Refined? Four Perspectives.

[5]  A. Chemero.  An Outline of a Theory of Affordances , 2003, How Shall Affordances be Refined? Four Perspectives.

[6]  T. Stoffregen.  Affordances as Properties of the Animal-Environment System , 2003, How Shall Affordances be Refined? Four Perspectives.

[7]  Kensuke Harada,et al.  Extracting grasping, contact points and objects motion from assembly demonstration , 2017, 2017 13th IEEE Conference on Automation Science and Engineering (CASE).

[8]  Nobutaka Shimada,et al.  Construction of Latent Descriptor Space and Inference Model of Hand-Object Interactions , 2017, IEICE Trans. Inf. Syst..

[9]  Giorgio Metta,et al.  Self-supervised learning of tool affordances from 3D tool representation through parallel SOM mapping , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Kensuke Harada,et al.  A framework for systematic accumulation, sharing and reuse of task implementation knowledge , 2016, 2016 IEEE/SICE International Symposium on System Integration (SII).

[11]  Jin-Hui Zhu,et al.  Affordance Research in Developmental Robotics: A Survey , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[12]  Nikolaos G. Tsagarakis,et al.  Detecting object affordances with Convolutional Neural Networks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Eren Erdal Aksoy,et al.  Towards a hierarchy of loco-manipulation affordances , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[14]  Florentin Wörgötter,et al.  Bootstrapping the Semantics of Tools: Affordance Analysis of Real World Objects on a Per-part Basis , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[15]  Frank Guerin,et al.  A model-based approach to finding substitute tools in 3D vision data , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Michael S. Ryoo,et al.  Learning Social Affordance for Human-Robot Interaction , 2016, IJCAI.

[17]  Hema Swetha Koppula,et al.  Anticipating Human Activities Using Object Affordances for Reactive Robotic Response , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Lea Fleischer,et al.  The Senses Considered As Perceptual Systems , 2016 .

[19]  Giorgio Metta,et al.  Multi-model approach based on 3D functional features for tool affordance learning in robotics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[20]  Oskar von Stryk,et al.  Achieving versatile manipulation tasks with unknown objects by supervised humanoid robots based on object templates , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[21]  Siddhartha S. Srinivasa,et al.  Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set , 2015, IEEE Robotics & Automation Magazine.

[22]  Ales Ude,et al.  Comparison of action-grounded and non-action-grounded 3-D shape features for object affordance classification , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[23]  Tamim Asfour,et al.  The KIT whole-body human motion database , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[24]  Rada Mihalcea,et al.  Mining semantic affordances of visual object categories , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Song-Chun Zhu,et al.  Understanding tools: Task-oriented object modeling, learning and recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Giorgio Metta,et al.  Self-supervised learning of grasp dependent tool affordances on the iCub Humanoid robot , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Máximo A. Roa,et al.  Functional power grasps transferred through warping and replanning , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Stephen Hart,et al.  The Affordance Template ROS package for robot task programming , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Yiannis Aloimonos,et al.  Affordance detection of tool parts from geometric features , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Karthik Mahesh Varadarajan,et al.  Topological mapping for robot navigation using affordance features , 2015, 2015 6th International Conference on Automation, Robotics and Applications (ICARA).

[31]  Joachim Hertzberg,et al.  The Role of Functional Affordances in Socializing Robots , 2015, International Journal of Social Robotics.

[32]  Aaron M. Dollar,et al.  The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments , 2015, Int. J. Robotics Res..

[33]  P. Abbeel,et al.  Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols , 2015, ArXiv.

[34]  Hedvig Kjellström,et al.  Functional Descriptors for Object Affordances , 2015 .

[35]  Justus H. Piater,et al.  Bootstrapping paired-object affordance learning with learned single-affordance features , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[36]  Koen V. Hindriks,et al.  Effective transfer learning of affordances for household robots , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[37]  Oskar von Stryk,et al.  Template-based manipulation in unstructured environments for supervised semi-autonomous humanoid robots , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[38]  Koen V. Hindriks,et al.  Active learning of affordances for robot use of household objects , 2014, 2014 IEEE-RAS International Conference on Humanoid Robots.

[39]  Akira Nakamura,et al.  Modeling of everyday objects for semantic grasp , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[40]  Luc De Raedt,et al.  Occluded object search by relational affordances , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[41]  Yukie Nagai,et al.  Parental scaffolding as a bootstrapping mechanism for learning grasp affordances and imitation skills , 2014, Robotica.

[42]  Alexandre Bernardino,et al.  Learning visual affordances of objects and tools through autonomous robot exploration , 2014, 2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[43]  Joachim Hertzberg,et al.  Finding Ways to Get the Job Done: An Affordance-Based Approach , 2014, ICAPS.

[44]  Danica Kragic,et al.  Data-Driven Grasp Synthesis—A Survey , 2013, IEEE Transactions on Robotics.

[45]  Tetsunari Inamura,et al.  Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools , 2013, Artificial Life and Robotics.

[46]  Koen V. Hindriks,et al.  Robot learning and use of affordances in goal-directed tasks , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[47]  Edwin Olson,et al.  Predicting object functionality using physical simulations , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[48]  Danica Kragic,et al.  Predicting slippage and learning manipulation affordances through Gaussian Process regression , 2013, 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids).

[49]  Markus Vincze,et al.  Parallel Deep Learning with Suggestive Activation for Object Category Recognition , 2013, ICVS.

[50]  Shaogang Ren,et al.  Human-object-object-interaction affordance , 2013, 2013 IEEE Workshop on Robot Vision (WORV).

[51]  Danica Kragic,et al.  Learning a dictionary of prototypical grasp-predicting parts from grasping experience , 2013, 2013 IEEE International Conference on Robotics and Automation.

[52]  Hedvig Kjellström,et al.  Functional object descriptors for human activity modeling , 2013, 2013 IEEE International Conference on Robotics and Automation.

[53]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[54]  Sinan Kalkan,et al.  Learning Social Affordances and Using Them for Planning , 2013, CogSci.

[55]  Markus Vincze,et al.  AfRob: The affordance network ontology for robots , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[56]  Markus Vincze,et al.  AfNet: The Affordance Network , 2012, Asian Conference on Computer Vision.

[57]  Luc De Raedt,et al.  Learning relational affordance models for robots in multi-object manipulation tasks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[58]  Danica Kragic,et al.  Generalizing grasps across partly similar objects , 2012, 2012 IEEE International Conference on Robotics and Automation.

[59]  R. Amant,et al.  Affordances for robots: a brief survey , 2012 .

[60]  Barbara Caputo,et al.  Using Object Affordances to Improve Object Recognition , 2011, IEEE Transactions on Autonomous Mental Development.

[61]  Aaron M. Dollar,et al.  Classifying human manipulation behavior , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[62]  Luc Van Gool,et al.  What makes a chair a chair? , 2011, CVPR 2011.

[63]  Alexei A. Efros,et al.  From 3D scene geometry to human workspace , 2011, CVPR 2011.

[64]  Ashutosh Saxena,et al.  Efficient grasping from RGBD images: Learning using a new rectangle representation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[65]  Danica Kragic,et al.  Visual object-action recognition: Inferring object affordances from human demonstration , 2011, Comput. Vis. Image Underst..

[66]  James M. Rehg,et al.  Affordance Prediction via Learned Object Attributes , 2011 .

[67]  Justus H. Piater,et al.  Refining grasp affordance models by experience , 2010, 2010 IEEE International Conference on Robotics and Automation.

[68]  James M. Rehg,et al.  Learning Visual Object Categories for Robot Affordance Prediction , 2010, Int. J. Robotics Res..

[69]  Danica Kragic,et al.  Exploring affordances in robot grasping through latent structure representation , 2010, ECCV 2010.

[70]  Anis Sahbani,et al.  A hybrid approach for grasping 3D objects , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[71]  Danica Kragic,et al.  Learning grasping affordance using probabilistic and ontological approaches , 2009, 2009 International Conference on Advanced Robotics.

[72]  N. Kruger,et al.  Learning object-specific grasp affordance densities , 2009, 2009 IEEE 8th International Conference on Development and Learning.

[73]  Andrew H. Fagg,et al.  Grasping Affordances: Learning to Connet vission to Hand Action , 2009 .

[74]  Garth Zeglin,et al.  Preparatory object rotation as a human-inspired grasping strategy , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[75]  Kohtaro Ohba,et al.  Learning affordance for semantic robots using ontology approach , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[76]  Kazuhiko Kawamura,et al.  Towards a cognitive robot that uses internal rehearsal to learn affordance relations , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[77]  Bernt Schiele,et al.  Functional Object Class Detection Based on Learned Affordance Cues , 2008, ICVS.

[78]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[79]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[80]  Maya Cakmak,et al.  To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control , 2007, Adapt. Behav..

[81]  Manuel Lopes,et al.  Affordance-based imitation learning in robots , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[82]  J. Santos-Victor,et al.  Affordances, development and imitation , 2007, 2007 IEEE 6th International Conference on Development and Learning.

[83]  Maya Cakmak,et al.  The learning and use of traversability affordance using range images on a mobile robot , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[84]  Peter K. Allen,et al.  Graspit! A versatile simulator for robotic grasping , 2004, IEEE Robotics & Automation Magazine.

[85]  明彦 小長谷,et al.  遺伝子ネットワークの推定 (データマイニング・ビジネスプロセスとこう繋げ) -- (データマイニングツール(各社,大学を含む)適用実例集!) , 2003 .

[86]  Ergun Bicici,et al.  Reasoning About the Functionality of Tools and Physical Artifacts , 2003 .

[87]  Mark Steedman,et al.  Plans, Affordances, And Combinatory Grammar , 2002 .

[88]  Ronald C. Arkin,et al.  An Behavior-based Robotics , 1998 .

[89]  M. Turvey.  Affordances and Prospective Control: An Outline of the Ontology , 1992 .

[90]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[91]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[92]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[93]  R. Shaw,et al.  Perceiving, Acting and Knowing : Toward an Ecological Psychology , 1978 .

[94]  J. J. Gibson.  The theory of affordances , 1977 .

[95]  P. L. Adams.  THE ORIGINS OF INTELLIGENCE IN CHILDREN , 1976 .

[96]  Richard Fikes,et al.  STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving , 1971, IJCAI.