A Partial Contour Similarity-Based Approach to Visual Affordances in Habile Agents

In a typical tool use task, we can view both the relationship between the agent and the tool and the relationship between the tool and the target in terms of affordances. One set of affordances relates to the ability of the agent to manipulate the tool, while a second set of affordances relates to the ability of the agent to manipulate the target by means of the tool. In both cases, effective tool use is facilitated by the coupling of one object to another: agent-to-tool-to-target. In this paper, we focus on the visual identification of such affordances via contour similarity. Objects with complementary contour segments can fit together, which suggests possible opportunities for effective interactions. We present a system for the identification and evaluation of partial contour-based matches and analyze the system’s behavior. We propose a set of sample tool-use scenarios as part of our analysis. We demonstrate the use of the system in providing guidance to an autonomous robotic agent performing tool selection tasks.

[1]  Harry Heft.  Ecological Psychology in Context: James Gibson, Roger Barker, and the Legacy of William James's Radical Empiricism , 2001 .

[2]  Jane Van Lawick-Goodall,et al.  Tool-Using in Primates and Other Vertebrates , 1971 .

[3]  E. Reed.  Encountering the world: Toward an ecological psychology. , 1997 .

[4]  T. Matsuzawa.  Primate Foundations of Human Intelligence: A View of Tool Use in Nonhuman Primates and Fossil Hominids , 2008 .

[5]  Harry Heft.  Affordances and the Body: An Intentional Analysis of Gibson's Ecological Approach to Visual Perception , 1989 .

[6]  R. Shepard,et al.  Mental Rotation of Three-Dimensional Objects , 1971, Science.

[7]  J. Sinapov,et al.  Detecting the functional similarities between tools using a hierarchical representation of outcomes , 2008, 2008 7th IEEE International Conference on Development and Learning.

[8]  J. Gyr.  Is a theory of direct visual perception adequate? , 1972, Psychological bulletin.

[9]  D. Povinelli.  Folk physics for apes , 2000 .

[10]  Elizabeth Bates,et al.  Perceptual aspects of tool using in infancy , 1980 .

[11]  Kin-Man Lam,et al.  Extraction of the Euclidean skeleton based on a connectivity criterion , 2003, Pattern Recognit..

[12]  Jitendra Malik,et al.  Matching Shapes , 2001, ICCV.

[13]  J. Fodor,et al.  How direct is visual perception?: Some reflections on Gibson's “ecological approach” , 1981, Cognition.

[14]  Ruzena Bajcsy,et al.  Active Investigation of Functionality , 1994 .

[15]  Harold W. Thimbleby,et al.  Affordance and Symmetry in User Interfaces , 2008, Comput. J..

[16]  A.B. Wood,et al.  Effective tool use in a habile agent , 2005, 2005 IEEE Design Symposium, Systems and Information Engineering.

[17]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[18]  Longin Jan Latecki,et al.  Convexity Rule for Shape Decomposition Based on Discrete Contour Evolution , 1999, Comput. Vis. Image Underst..

[19]  Xin Li,et al.  2D Shape Decomposition Based on Combined Skeleton-Boundary Features , 2008, ISVC.

[20]  David S. Touretzky,et al.  Tekkotsu: A Framework for AIBO Cognitive Robotics , 2005, AAAI.

[21]  Louise Stark,et al.  Exploiting Context in Function-Based Reasoning , 2000, Sensor Based Intelligent Robots.

[22]  T. Stoffregen.  Affordances as Properties of the Animal-Environment System , 2003, How Shall Affordances be Refined? Four Perspectives.

[23]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[24]  Michelle Y. Merrill,et al.  Orangutan Cultures and the Evolution of Material Culture , 2003, Science.

[25]  Michael T. Turvey,et al.  Gibsonian Affordances for Roboticists , 2007, Adapt. Behav..

[26]  Anne Kuefer,et al.  The Case For Mental Imagery , 2016 .

[27]  Thomas A. Stoffregen,et al.  Affordances and Events , 2000 .

[28]  Ingemar Ragnemalm,et al.  The Euclidean distance transform in arbitrary dimensions , 1992, Pattern Recognit. Lett..

[29]  Song-Chun Zhu,et al.  Understanding tools: Task-oriented object modeling, learning and recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Jitendra Malik,et al.  Shape Context: A New Descriptor for Shape Matching and Object Recognition , 2000, NIPS.

[31]  Manish Singh,et al.  Geometric determinants of shape segmentation: Tests using segment identification , 2007, Vision Research.

[32]  Giorgio Metta,et al.  Multi-model approach based on 3D functional features for tool affordance learning in robotics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[33]  J. Alcock.  THE EVOLUTION OF THE USE OF TOOLS BY FEEDING ANIMALS , 1972, Evolution; international journal of organic evolution.

[34]  Harold W. Thimbleby.  Reflections on symmetry , 2002, AVI '02.

[35]  A. Whiten,et al.  Cultures in chimpanzees , 1999, Nature.

[36]  S. Greenberg,et al.  The Psychology of Everyday Things , 2012 .

[37]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[38]  Donald A. Norman,et al.  Affordance, conventions, and design , 1999, INTR.

[39]  M. Dogar,et al.  Afford or Not to Afford : A New Formalization of Affordances Toward Affordance-Based Robot , 2007 .

[40]  Wendell H. Oswalt.  Habitat and technology;: The evolution of hunting , 1972 .

[41]  W H Warren,et al.  Perceiving affordances: visual guidance of stair climbing. , 1984, Journal of experimental psychology. Human perception and performance.

[42]  Harold W. Thimbleby.  Affordance and Symmetry , 2001, DSV-IS.

[43]  Harry Heft.  Affordances, Dynamic Experience, and the Challenge of Reification , 2003, How Shall Affordances be Refined? Four Perspectives.

[44]  Remco C. Veltkamp,et al.  State of the Art in Shape Matching , 2001, Principles of Visual Information Retrieval.

[45]  A. Chemero.  An Outline of a Theory of Affordances , 2003, How Shall Affordances be Refined? Four Perspectives.

[46]  Nicholas P. Holmes,et al.  The body schema and multisensory representation(s) of peripersonal space , 2004, Cognitive Processing.

[47]  Ruzena Bajcsy,et al.  Interactive Recognition and Representation of Functionality , 1995, Comput. Vis. Image Underst..

[48]  Alexander Stoytchev,et al.  Behavior-Grounded Representation of Tool Affordances , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[49]  R. Amant,et al.  Revisiting the definition of animal tool use , 2008, Animal Behaviour.

[50]  Endre E. Kadar,et al.  Toward an Ecological Field Theory of Perceptual Control of Locomotion , 2000 .

[51]  Remco C. Veltkamp,et al.  Shape Similarity Measures, Properties and Constructions , 2000, VISUAL.

[52]  P. Danielsson.  Euclidean distance mapping , 1980 .

[53]  Joanna McGrenere,et al.  Affordances: Clarifying and Evolving a Concep , 2000, Graphics Interface.

[54]  Robin R. Murphy,et al.  Case studies of applying Gibson's ecological approach to mobile robots , 1999, IEEE Trans. Syst. Man Cybern. Part A.

[55]  T. Stoffregen.  Breadth and Limits of the Affordance Concept , 2004 .

[56]  G. Hunt.  Manufacture and use of hook-tools by New Caledonian crows , 1996, Nature.

[57]  S. Kosslyn.  Image and Brain: The Resolution of the Imagery Debate , 1994, Journal of Cognitive Neuroscience.

[58]  Keith S. Jones,et al.  What Is an Affordance? , 2003, How Shall Affordances be Refined? Four Perspectives.

[59]  Trevor Darrell,et al.  Object Recognition using Locality Sensitive Hashing of Shape Contexts , 2006 .

[60]  Connor Schenck,et al.  Interactive object recognition using proprioceptive and auditory feedback , 2011, Int. J. Robotics Res..

[61]  A. Noē.  Direct Perception , 2022 .

[62]  Lea Fleischer,et al.  The Senses Considered As Perceptual Systems , 2016 .

[63]  Luciano da Fontoura Costa,et al.  2D Euclidean distance transform algorithms: A comparative survey , 2008, CSUR.

[64]  C. Michaels.  Affordances: Four Points of Debate , 2003, How Shall Affordances be Refined? Four Perspectives.

[65]  D. Papadias,et al.  Computational Imagery , 1992, Cogn. Sci..

[66]  J. Stillwell,et al.  Symmetry , 2000, Am. Math. Mon..

[67]  Charles C. Kemp,et al.  Visual Tool Tip Detection and Position Estimation for Robotic Manipulation of Unknown Human Tools , 2005 .

[68]  S. Ullman.  Against direct perception , 1980, Behavioral and Brain Sciences.

[69]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[70]  Robert St. Amant,et al.  Tool Use for Autonomous Agents , 2005, AAAI.

[71]  William W. Gaver.  Technology affordances , 1991, CHI.

[72]  Tetsuya Ogata,et al.  Tool-Body Assimilation Model Based on Body Babbling and Neurodynamical System , 2015 .

[73]  G. Rizzolatti,et al.  Premotor cortex and the recognition of motor actions. , 1996, Brain research. Cognitive brain research.

[74]  P. Cesari,et al.  A common perceptual parameter for stair climbing for children, young and old adults. , 2003, Human movement science.

[75]  Tetsunari Inamura,et al.  Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools , 2013, Artificial Life and Robotics.

[76]  Wenyu Liu,et al.  Skeleton Pruning by Contour Partitioning with Discrete Curve Evolution , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Harold W. Thimbleby,et al.  Symmetry for successful interactive systems , 2002, CHINZ '02.

[78]  W. Warren,et al.  Visual guidance of walking through apertures: body-scaled information for affordances. , 1987, Journal of experimental psychology. Human perception and performance.

[79]  H. Barlow.  Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[80]  M. Tanaka,et al.  Coding of modified body schema during tool use by macaque postcentral neurones. , 1996, Neuroreport.

[81]  Marshall W. Bern,et al.  A global approach to automatic solution of jigsaw puzzles , 2002, SCG '02.

[82]  Jitendra Malik,et al.  Shape contexts enable efficient retrieval of similar shapes , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[83]  Alexandre Bernardino,et al.  Learning intermediate object affordances: Towards the development of a tool concept , 2014, 4th International Conference on Development and Learning and on Epigenetic Robotics.

[84]  Herbert A. Simon,et al.  Situated Action: A Symbolic Interpretation , 1993, Cogn. Sci..

[85]  Tetsuro Matsuzawa,et al.  Nesting cups and metatools in chimpanzees , 1991, Behavioral and Brain Sciences.

[86]  Kevin W. Bowyer,et al.  Function from visual analysis and physical interaction: a methodology for recognition of generic classes of objects , 1998, Image Vis. Comput..

[87]  Atsushi Iriki,et al.  Tool-Use Training in a Species of Rodent: The Emergence of an Optimal Motor Strategy and Functional Understanding , 2008, PloS one.

[88]  M. Turvey.  Affordances and Prospective Control: An Outline of the Ontology , 1992 .

[89]  Alex Kirlik,et al.  On Stoffregen's Definition of Affordances , 2004 .

[90]  Lloyd Carter Williams.  Dynamic Ontology Driven Learning and Control of Robotic Tool Using Behavior , 2009 .

[91]  Luca Bogoni,et al.  More than just shape: a representation for functionality , 1998, Artif. Intell. Eng..