Interactive object recognition using proprioceptive and auditory feedback

In this paper we propose a method for interactive recognition of household objects using proprioceptive and auditory feedback. In our experiments, the robot observed the changes in its proprioceptive and auditory sensory streams while performing five exploratory behaviors (lift, shake, drop, crush, and push) on 50 common household objects (e.g. bottles, cups, balls, toys, etc.). The robot was tasked with recognizing the objects it was manipulating by feeling them and listening to the sounds that they make without using any visual information. The results show that both proprioception and audio, coupled with exploratory behaviors, can be used successfully for object recognition. Furthermore, the robot was able to integrate feedback from the two modalities, to achieve even better recognition accuracy. Finally, the results show that the robot can boost its recognition rate even further by applying multiple different exploratory behaviors on the object.

[1]  S. B. Needleman,et al.  A general method applicable to the search for similarities in the amino acid sequence of two proteins. , 1970, Journal of molecular biology.

[2]  Christopher G. Atkeson,et al.  Estimation of Inertial Parameters of Manipulator Loads and Links , 1986 .

[3]  Hsiao-Wuen Hon,et al.  An overview of the SPHINX speech recognition system , 1990, IEEE Trans. Acoust. Speech Signal Process..

[4]  M. Heller.  Haptic Dominance in Form Perception: Vision versus Proprioception , 1992, Perception.

[5]  William W. Gaver.  What in the World Do We Hear? An Ecological Approach to Auditory Event Perception , 1993 .

[6]  Eric Krotkov,et al.  Robotic Perception of Material: Experiments with Shape-Invariant Acoustic Measures of Material Type , 1995, ISER.

[7]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[8]  John M. Hollerbach,et al.  The Calibration Index and Taxonomy for Robot Kinematic Calibration Methods , 1996, Int. J. Robotics Res..

[9]  K. Pribram,et al.  Learning as self-organization , 1996 .

[10]  Oussama Khatib,et al.  Experimental Robotics IV, The 4th International Symposium, Stanford, California, USA, June 30 - July 2, 1995 , 1997, ISER.

[11]  M. Krabbes,et al.  Modelling of robot dynamics based on a multi-dimensional RBF-like neural network , 1999, Proceedings 1999 International Conference on Information Intelligence and Systems (Cat. No.PR00446).

[12]  T. Power.  Play and Exploration in Children and Animals , 1999 .

[13]  Kiyotaka Izumi,et al.  Evolving Runge-Kutta-Gill RBF networks to estimate the dynamics of a multi-link manipulator , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[14]  Dinesh K. Pai,et al.  Active measurement of contact sounds , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[15]  D. Muir,et al.  Three-year-olds' difficulty with the appearance--reality distinction: is it real or is it apparent? , 2000, Developmental psychology.

[16]  Joshua L. Richmond,et al.  Automatic measurement and modelling of contact sounds , 2000 .

[17]  Gonzalo Navarro,et al.  A guided tour to approximate string matching , 2001, CSUR.

[18]  A. Chan,et al.  Growing hierarchical self organising map (GHSOM) toolbox: visualisations and enhancements , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[19]  Giorgio Metta,et al.  Early integration of vision and manipulation , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[20]  Giorgio Metta,et al.  Better Vision through Manipulation , 2003, Adapt. Behav..

[21]  L. Natale,et al.  Learning haptic representation of objects , 2004 .

[22]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[23]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[24]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[25]  Andrew W. Moore,et al.  Locally Weighted Learning , 1997, Artificial Intelligence Review.

[26]  Lorenzo Natale,et al.  Tapping into Touch , 2005 .

[27]  M. Grassi.  Do we hear size or sound? Balls dropped on plates , 2005, Perception & psychophysics.

[28]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[29]  Richard Alan Peters,et al.  Sensory-Motor Manifold Structure Induced by Task Outcome: Experiments with Robonaut , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[30]  Bruno L. Giordano,et al.  Material identification of real impact sounds: effects of size variation in steel, glass, wood, and plexiglass plates. , 2006, The Journal of the Acoustical Society of America.

[31]  Friedrich M. Wahl,et al.  On-line rigid object recognition and pose estimation based on inertial parameters , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Tomoaki Nakamura,et al.  Multimodal object categorization by a robot , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  Andrew Y. Ng,et al.  STAIR: Hardware and Software Architecture , 2007 .

[34]  Nico Blodow,et al.  Towards 3D Point cloud based object maps for household environments , 2008, Robotics Auton. Syst..

[35]  Jivko Sinapov,et al.  Interactive learning of the acoustic properties of household objects , 2009, 2009 IEEE International Conference on Robotics and Automation.

[36]  C. Schenck,et al.  Interactive Object Recognition Using Proprioceptive Feedback , 2009 .

[37]  Jivko Sinapov,et al.  From Acoustic Object Recognition to Object Categorization by a Humanoid Robot , 2009 .

[38]  Dermot Lynott,et al.  Modality exclusivity norms for 423 object properties , 2009, Behavior research methods.

[39]  Danica Kragic,et al.  An Active Vision System for Detecting, Fixating and Manipulating Objects in the Real World , 2010, Int. J. Robotics Res..

[40]  Geoffrey A. Hollinger,et al.  HERB: a home exploring robotic butler , 2010, Auton. Robots.

[41]  Jivko Sinapov,et al.  The Boosting Effect of Exploratory Behaviors , 2010, AAAI.

[42]  Ana I. González Acuña.  An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization , 2012 .

[43]  Christus,et al.  A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins , 2022 .