Towards Grasp-Oriented Visual Perception for Humanoid Robots

A distinct property of robot vision systems is that they are embodied. Visual information is extracted for the purpose of moving in and interacting with the environment. Thus, different types of perception-action cycles need to be implemented and evaluated. In this paper, we study the problem of designing a vision system for the purpose of object grasping in everyday environments. This vision system is firstly targeted at the interaction with the world through recognition and grasping of objects and secondly at being an interface for the reasoning and planning module to the real world. The latter provides the vision system with a certain task that drives it and defines a specific context, i.e. search for or identify a certain object and analyze it for potential later manipulation. We deal with cases of: (i) known objects, (ii) objects similar to already known objects, and (iii) unknown objects. The perception-action cycle is connected to the reasoning system based on the idea of affordances. All three cases are also related to the state of the art and the terminology in the neuroscientific area.

[1]  David Ingle,et al.  Locating and identifying: Two modes of visual processing , 1967 .

[2]  R. Shaw,et al.  Perceiving, Acting and Knowing : Toward an Ecological Psychology , 1978 .

[3]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[4]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[5]  D. Pandya,et al.  Projections to the frontal cortex from the posterior parietal region in the rhesus monkey , 1984, The Journal of comparative neurology.

[6]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[7]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[8]  M. A. Fischler,et al.  Context-based vision: Recognition of natural scenes , 1989, Twenty-Third Asilomar Conference on Signals, Systems and Computers, 1989..

[9]  Andrew K. C. Wong,et al.  A gray-level threshold selection method based on maximum entropy principle , 1989, IEEE Trans. Syst. Man Cybern..

[10]  P. Goldman-Rakic,et al.  Connections of the ventral granular frontal cortex of macaques with perisylvian premotor and somatosensory areas: Anatomical evidence for somatic representation in primate frontal association cortex , 1989, The Journal of comparative neurology.

[11]  John K. Tsotsos.  The Complexity of Perceptual Search Tasks , 1989, IJCAI.

[12]  John K. Tsotsos.  Analyzing vision at the complexity level , 1990, Behavioral and Brain Sciences.

[13]  M. Jeannerod,et al.  Measuring time to awareness , 1991, Neuroreport.

[14]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[15]  D. V. van Essen,et al.  A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[16]  J. B. Preston,et al.  Interconnections between the prefrontal cortex and the premotor areas in the frontal lobe , 1994, The Journal of comparative neurology.

[17]  H. Bülthoff,et al.  Separate neural pathways for the visual analysis of object shape in perception and prehension , 1994, Current Biology.

[18]  Leslie G. Ungerleider,et al.  Connections of inferior temporal areas TEO and TE with parietal and frontal cortex in macaque monkeys. , 1994, Cerebral cortex.

[19]  Jude W. Shavlik,et al.  in Advances in Neural Information Processing , 1996 .

[20]  S. Ullman,et al.  Spatial Context in Recognition , 1996, Perception.

[21]  Karun B. Shimoga,et al.  Robot Grasp Synthesis Algorithms: A Survey , 1996, Int. J. Robotics Res..

[22]  Michael A. Arbib,et al.  Modeling parietal-premotor interactions in primate control of grasping , 1998, Neural Networks.

[23]  R. Ellis,et al.  On the relations between seen objects and components of potential actions. , 1998, Journal of experimental psychology. Human perception and performance.

[24]  G. Rizzolatti,et al.  The organization of the cortical motor system: new concepts. , 1998, Electroencephalography and clinical neurophysiology.

[25]  Sariel Har-Peled,et al.  Efficiently approximating the minimum-volume bounding box of a point set in three dimensions , 1999, SODA '99.

[26]  Allison M. Okamura,et al.  An overview of dexterous manipulation , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[27]  H. Sakata,et al.  Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area AIP. , 2000, Journal of neurophysiology.

[28]  William Townsend,et al.  The BarrettHand grasper – programmably flexible part handling and assembly , 2000 .

[29]  C. Koch,et al.  Models of bottom-up and top-down visual attention , 2000 .

[30]  Sarah H. Creem,et al.  Grasping objects by their handles: a necessary interaction between cognition and action. , 2001, Journal of experimental psychology. Human perception and performance.

[31]  Sunghee Choi,et al.  The power crust , 2001, SMA '01.

[32]  G. Rizzolatti,et al.  The Cortical Motor System , 2001, Neuron.

[33]  Antonio Torralba,et al.  Statistical Context Priming for Object Detection , 2001, ICCV.

[34]  Jun Saiki,et al.  Stochastic Guided Search Model for Search Asymmetries in Visual Search Tasks , 2002, Biologically Motivated Computer Vision.

[35]  Ravi S. Menon,et al.  Differential Effects of Viewpoint on Object-Driven Activation in Dorsal and Ventral Streams , 2002, Neuron.

[36]  J. Decety,et al.  Does visual perception of object afford action? Evidence from a neuroimaging study , 2002, Neuropsychologia.

[37]  Zhaoping Li.  A saliency map in primary visual cortex , 2002, Trends in Cognitive Sciences.

[38]  M. Gentilucci.  Object motor representation and reaching–grasping control , 2002, Neuropsychologia.

[39]  Helge J. Ritter,et al.  Multi-modal human-machine communication for instructing robot grasping tasks , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  Henrik I. Christensen,et al.  Automatic grasp planning using shape primitives , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[41]  Antonio Torralba,et al.  Top-down control of visual attention in object detection , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[42]  Gerd Hirzinger,et al.  Grasping the dice by dicing the grasp , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[43]  Dimitrios Katsoulas,et al.  Reliable recovery of piled box-like objects via parabolically deformable superquadrics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[44]  Atilla Baskurt,et al.  Segmentation and Superquadric Modeling of 3D Objects , 2003, WSCG.

[45]  B. Draper,et al.  Evaluation of Selective Attention under Similarity Transforms , 2003 .

[46]  John K. Tsotsos.  A ‘complexity level’ analysis of immediate vision , 2004, International Journal of Computer Vision.

[47]  Xing Xie,et al.  Salient Region Detection Using Weighted Feature Maps Based on the Human Visual Attention Model , 2004, PCM.

[48]  Antonio Morales,et al.  Using Experience for Assessing Grasp Reliability , 2004, Int. J. Humanoid Robotics.

[49]  Gerd Hirzinger,et al.  Grasp planning: how to choose a suitable task wrench space , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[50]  Sangmoon Choi.  Biologically Motivated Visual Attention System Using Bottom-up Saliency Map and Top-down Inhibition , 2004 .

[51]  H. Christensen,et al.  Object detection using background context , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[52]  Antonio Morales Escrig.  Learning to predict grasp reliability with a multifinger robot hand by using visual features , 2004 .

[53]  Jan-Olof Eklundh,et al.  Attending, Foveating and Recognizing Objects in Real World Scenes , 2004, BMVC.

[54]  G. Rizzolatti,et al.  Localization of grasp representations in humans by PET: 1. Observation versus execution , 1996, Experimental Brain Research.

[55]  G. Rizzolatti,et al.  Cortico-cortical connections of two electrophysiologically identified arm representations in the mesial agranular frontal cortex , 2004, Experimental Brain Research.

[56]  R. S. Johansson,et al.  Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects , 2004, Experimental Brain Research.

[57]  M. Bar.  Visual objects in context , 2004, Nature Reviews Neuroscience.

[58]  Eli Brenner,et al.  On the relation between object shape and grasping kinematics. , 2004, Journal of neurophysiology.

[59]  Peter K. Allen,et al.  Graspit! A versatile simulator for robotic grasping , 2004, IEEE Robotics & Automation Magazine.

[60]  R. Alami,et al.  Grasp Planning for Non-Convex Objects , 2005 .

[61]  Rolf A. Zwaan,et al.  Grounding Cognition: Introduction to Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking , 2005 .

[62]  Cynthia Breazeal,et al.  Action parsing and goal inference using self as simulator , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[63]  Bruce A. Draper,et al.  Evaluation of selective attention under similarity transformations , 2005, Comput. Vis. Image Underst..

[64]  Guido Bugmann,et al.  Vision-based urban navigation procedures for verbally instructed robots , 2005, Robotics Auton. Syst..

[65]  L. Fogassi,et al.  Functional properties of grasping-related neurons in the ventral premotor area F5 of the macaque monkey. , 2006, Journal of neurophysiology.

[66]  Jan-Olof Eklundh,et al.  The Eigen-Transform and Applications , 2006, ACCV.

[67]  Simone Frintrop,et al.  VOCUS: A Visual Attention System for Object Detection and Goal-Directed Search , 2006, Lecture Notes in Computer Science.

[68]  Efrain Lopez Damian.  Grasp planning for object manipulation by an autonomous robot , 2006 .

[69]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects , 2006, NIPS.

[70]  Tamim Asfour,et al.  ARMAR-III: An Integrated Humanoid Platform for Sensory-Motor Control , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[71]  Gordon Cheng,et al.  Foveated vision systems with two cameras per eye , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[72]  Paul Ciprian Patic,et al.  THE BARRETTHAND GRASPER - PROGRAMMABLY FLEXIBLE PART HANDLING AND ASSEMBLY , 2006 .

[73]  Christopher W. Geib,et al.  Object Action Complexes as an Interface for Planning and Robot Control , 2006 .

[74]  Marc Alexa,et al.  To appear in the ACM SIGGRAPH conference proceedings FiberMesh : Designing Freeform Surfaces with 3 D Curves , 2007 .

[75]  José Santos-Victor,et al.  A Developmental Roadmap for Learning by Imitation in Robots , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[76]  Markus Vincze,et al.  Efficient 3D Object Detection by Fitting Superquadrics to Range Image Data for Robot's Object Manipulation , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[77]  P. Allen,et al.  Dexterous Grasping via Eigengrasps : A Low-dimensional Approach to a High-complexity Problem , 2007 .

[78]  Danica Kragic,et al.  Learning and Evaluation of the Approach Vector for Automatic Grasp Generation and Planning , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[79]  Andrea Vedaldi,et al.  Objects in Context , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[80]  Bart Baesens,et al.  Comprehensible Credit Scoring Models Using Rule Extraction from Support Vector Machines , 2007, Eur. J. Oper. Res..

[81]  Marc Alexa,et al.  FiberMesh: designing freeform surfaces with 3D curves , 2007, SIGGRAPH 2007.

[82]  Jan-Olof Eklundh,et al.  An Attentional System Combining Top-Down and Bottom-Up Influences , 2008, WAPCV.

[83]  Danica Kragic,et al.  Early reactive grasping with second order 3D feature relations , 2007 .

[84]  Wolfram Burgard,et al.  Instace-Based AMN Classification for Improved Object Recognition in 2D and 3D Laser Range Data , 2007, IJCAI.

[85]  Dirk Kraft,et al.  An anthropomorphic grasping approach for an assistant humanoid robot , 2007 .

[86]  Peter K. Allen,et al.  Grasp Planning via Decomposition Trees , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[87]  Eris Chinellato,et al.  Neural Coding in the Dorsal Visual Stream , 2008, SAB.

[88]  三嶋 博之.  The theory of affordances , 2008 .

[89]  Manuel Lopes,et al.  Learning Object Affordances: From Sensory--Motor Coordination to Imitation , 2008, IEEE Transactions on Robotics.

[90]  Antonio Morales,et al.  Vision-based grasp planning of 3D objects by extending 2D contour based algorithms , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[91]  Bernt Schiele,et al.  Functional Object Class Detection Based on Learned Affordance Cues , 2008, ICVS.

[92]  Lawson L. S. Wong,et al.  Learning Grasp Strategies with Partial Shape Information , 2008, AAAI.

[93]  Anis Sahbani,et al.  Handling Objects by Their Handles , 2008 .

[94]  Danica Kragic,et al.  Selection of robot pre-grasps using box-based shape approximation , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[95]  Nico Blodow,et al.  Functional object mapping of kitchen environments , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[96]  Danica Kragic,et al.  Minimum volume bounding box decomposition for shape approximation in robot grasping , 2008, 2008 IEEE International Conference on Robotics and Automation.

[97]  Gordon Cheng,et al.  Biologically Based Top-Down Attention Modulation for Humanoid Interactions , 2008, Int. J. Humanoid Robotics.

[98]  Philippe Martinet,et al.  Biologically-inspired 3D grasp synthesis based on visual exploration , 2008, Auton. Robots.

[99]  Danica Kragic,et al.  Integration of Visual and Shape Attributes for Object Action Complexes , 2008, ICVS.

[100]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[101]  Danica Kragic,et al.  Demonstration-based learning and control for automatic grasping , 2009, Intell. Serv. Robotics.

[102]  Danica Kragic,et al.  Learning of 2D grasping strategies from box-based 3D object approximations , 2009, Robotics: Science and Systems.

[103]  Danica Kragic,et al.  Grasping familiar objects using shape context , 2009, 2009 International Conference on Advanced Robotics.

[104]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.