Value systems for developmental cognitive robotics: A survey

This paper surveys value systems for developmental cognitive robotics. A value system permits a biological brain to increase the likelihood of neural responses to selected external phenomena. Many machine learning algorithms capture the essence of this learning process. However, computational value systems aim not only to support learning, but also autonomous attention focus to direct learning. This combination of unsupervised attention focus and learning aims to address the grand challenge of autonomous mental development for machines. This survey examines existing value systems for developmental cognitive robotics in this context. We examine the definitions of value used-including recent pioneering work in intrinsic motivation as value-as well as initialisation strategies for innate values, update strategies for acquired value and the data structures used for storing value. We examine the extent to which existing value systems support attention focus, learning and prediction in an unsupervised setting. The types of robots and applications in which these value systems are used are also examined, as well as the ways that these applications are evaluated. Finally, we study the strengths and limitations of current value systems for developmental cognitive robots and conclude with a set of research challenges for this field.

[1]  Giulio Sandini,et al.  Developmental robotics: a survey , 2003, Connect. Sci..

[2]  Helmut Hauser,et al.  Self-exploration of the Stumpy Robot with Predictive Information Maximization , 2014, SAB.

[3]  Marco Mirolli,et al.  Which is the best intrinsic motivation signal for learning multiple skills? , 2013, Front. Neurorobot..

[4]  Pierre-Yves Oudeyer,et al.  Active learning of inverse models with intrinsically motivated goal exploration in robots , 2013, Robotics Auton. Syst..

[5]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[6]  Mark D. Humphries,et al.  A robot model of the basal ganglia: Behavior and intrinsic processing , 2006, Neural Networks.

[7]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[8]  Pierre-Yves Oudeyer,et al.  Behavioral Diversity Generation in Autonomous Exploration through Reuse of Past Experience , 2016, Front. Robot. AI.

[9]  Jeffrey L. Krichmar,et al.  Neuromorphic and Brain-Based Robots: Neuromorphic robots: biologically and neurally inspired designs , 2011 .

[10]  Niclas Bergman,et al.  Recursive Bayesian Estimation : Navigation and Tracking Applications , 1999 .

[11]  Pierre-Yves Oudeyer,et al.  Maturationally-constrained competence-based intrinsically motivated learning , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[12]  Silvia Rossi,et al.  The role of intrinsic motivations in attention allocation and shifting , 2014, Front. Psychol..

[13]  Michael Lardelli,et al.  The Guinea Pig as a Model for Sporadic Alzheimer’s Disease (AD): The Impact of Cholesterol Intake on Expression of AD-Related Genes , 2013, PloS one.

[14]  Yibin Li,et al.  Could Cloud Technology be Useful in Autonomous Mental Developmental Robotics? A Case Study , 2016, Int. J. Robotics Autom..

[15]  Kathryn E. Merrick,et al.  Intrinsically motivated particle swarm optimisation applied to task allocation for workplace hazard detection , 2016, Adapt. Behav..

[16]  F. Kaplan,et al.  The challenges of joint attention , 2006 .

[17]  Karl J. Friston,et al.  What is value—accumulated reward or evidence? , 2012, Front. Neurorobot..

[18]  Daniele P. Radicioni,et al.  From human to artificial cognition and back: New perspectives on cognitively inspired AI systems , 2015, Cognitive Systems Research.

[19]  Masaki Ogino,et al.  Cognitive Developmental Robotics: A Survey , 2009, IEEE Transactions on Autonomous Mental Development.

[20]  Ehud Ahissar,et al.  A Curious Emergence of Reaching , 2012, TAROS.

[21]  Rafael Pérez y Pérez,et al.  Dev E-R: A computational model of early cognitive development as a creative process , 2015, Cognitive Systems Research.

[22]  Raymond J. Dolan,et al.  Keep focussing: striatal dopamine multiple functions resolved in a single mechanism tested in a simulated humanoid robot , 2014, Front. Psychol..

[23]  Jürgen Schmidhuber,et al.  Learning tactile skills through curious exploration , 2012, Front. Neurorobot..

[24]  Richard L. Lewis,et al.  Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective , 2010, IEEE Transactions on Autonomous Mental Development.

[25]  Marco Mirolli,et al.  Intrinsically Motivated Learning in Natural and Artificial Systems , 2013 .

[26]  Marco Mirolli,et al.  GRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[27]  S. Schaal,et al.  Robot juggling: implementation of memory-based learning , 1994, IEEE Control Systems.

[28]  P. Rakic.  Progress: Neurogenesis in adult primate neocortex: an evaluation of the evidence , 2002, Nature Reviews Neuroscience.

[29]  Xiao Huang,et al.  Novelty and Reinforcement Learning in the Value System of Developmental Robots , 2002 .

[30]  Kevin N. Gurney,et al.  A biologically plausible embodied model of action discovery , 2012, Front. Neurorobot..

[31]  Jörg Conradt,et al.  Serendipitous Offline Learning in a Neuromorphic Robot , 2016, Front. Neurorobot..

[32]  James L. McClelland,et al.  Autonomous Mental Development by Robots and Animals , 2001, Science.

[33]  Jeffrey L. Krichmar,et al.  Brain-based devices: intelligent systems based on principles of the nervous system , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[34]  Olivier Sigaud,et al.  Towards Deep Developmental Learning , 2016, IEEE Transactions on Cognitive and Developmental Systems.

[35]  Olaf Sporns,et al.  Plasticity in Value Systems and its Role in Adaptive Behavior , 2000, Adapt. Behav..

[36]  Jürgen Leitner,et al.  The Modular Behavioral Environment for Humanoids and other Robots (MoBeE) , 2012, ICINCO.

[37]  Kathryn E. Merrick,et al.  Modeling Behavior Cycles as a Value System for Developmental Robots , 2010, Adapt. Behav..

[38]  Jürgen Schmidhuber,et al.  An intrinsic value system for developing multiple invariant representations with incremental slowness learning , 2013, Front. Neurorobot..

[39]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[40]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[41]  Jürgen Leitner,et al.  Curiosity driven reinforcement learning for motion planning on humanoids , 2014, Front. Neurorobot..

[42]  Avelino J. Gonzalez,et al.  A Framework for the Qualitative Comparison of Diverse Developmental Agents , 2013, FLAIRS Conference.

[43]  Antoine Cully,et al.  Robots that can adapt like animals , 2014, Nature.

[44]  Karl J. Friston,et al.  Value-dependent selection in the brain: Simulation in a synthetic neural model , 1994, Neuroscience.

[45]  Hiroshi Ishiguro,et al.  Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[46]  Ehud Ahissar,et al.  Hierarchical curiosity loops and active sensing , 2012, Neural Networks.

[47]  Daniele P. Radicioni,et al.  From human to artificial cognition (and back): New perspectives on cognitively inspired AI systems , 2015, Cognitive Systems Research.

[48]  Kenji Doya,et al.  The Cyber Rodent Project: Exploration of Adaptive Mechanisms for Self-Preservation and Self-Reproduction , 2005, Adapt. Behav..

[49]  Jeffrey L. Krichmar,et al.  Neuromodulation as a robot controller , 2009, IEEE Robotics & Automation Magazine.

[50]  Steve B. Furber,et al.  Neural Systems Engineering , 2008, Computational Intelligence: A Compendium.

[51]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[52]  Ralf Der,et al.  Information Driven Self-Organization of Complex Robotic Behaviors , 2013, PloS one.

[53]  Kathryn E. Merrick,et al.  Achievement, affiliation, and power: Motive profiles for artificial agents , 2011, Adapt. Behav..

[54]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[55]  Jürgen Schmidhuber,et al.  Learning skills from play: Artificial curiosity on a Katana robot arm , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[56]  Olaf Sporns,et al.  Value and Self-Referential Control: Necessary Ingredients for the Autonomous Development of Flexible Intelligence , 2010 .

[57]  L. Merabet,et al.  The plastic human brain cortex. , 2005, Annual review of neuroscience.

[58]  Alexander Stoytchev,et al.  Some Basic Principles of Developmental Robotics , 2009, IEEE Transactions on Autonomous Mental Development.

[59]  Kathryn E. Merrick,et al.  A Comparative Study of Value Systems for Self-Motivated Exploration and Learning by Robots , 2010, IEEE Transactions on Autonomous Mental Development.

[60]  Stephen R. Marsland,et al.  A Real-Time Novelty Detector for a Mobile Robot , 2000, ArXiv.

[61]  W. Schultz.  Multiple reward signals in the brain , 2000, Nature Reviews Neuroscience.

[62]  Minoru Asada,et al.  Cognitive developmental robotics as a new paradigm for the design of humanoid robots , 2001, Robotics Auton. Syst..

[63]  Patricia Shaw,et al.  A psychology based approach for longitudinal development in cognitive robotics , 2014, Front. Neurorobot..

[64]  Jeffrey L. Krichmar,et al.  A neurorobotic platform to test the influence of neuromodulatory signaling on anxious and curious behavior , 2013, Front. Neurorobot..

[65]  Roderic A. Grupen,et al.  Error detection and surprise in stochastic robot actions , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[66]  A. Barto,et al.  Intrinsic motivations and open-ended development in animals, humans, and robots: an overview , 2014, Front. Psychol..

[67]  Xin Xu,et al.  Least Squares Policy Iteration Based on Random Vector Basis , 2015 .

[68]  Pierre-Yves Oudeyer,et al.  What is Intrinsic Motivation? A Typology of Computational Approaches , 2007, Frontiers Neurorobotics.

[69]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[70]  G. Edelman,et al.  Behavioral constraints in the development of neuronal properties: a cortical model embedded in a real-world device. , 1998, Cerebral cortex.

[71]  Mark H. Lee,et al.  Staged Competence Learning in Developmental Robotics , 2007, Adapt. Behav..

[72]  F. Karray,et al.  Computational Intelligence Techniques in Bio-inspired Robotics , 2009 .

[73]  Jeffrey L. Krichmar,et al.  as a Robot Controller Neuromodulation as a Robot Controller A Brain-Inspired Strategy for Controlling Autonomous Robots , 2009 .

[74]  Chu Kiong Loo,et al.  Incremental episodic segmentation and imitative learning of humanoid robot through self-exploration , 2016, Neurocomputing.

[75]  Scott Bolland,et al.  Exploring the Periphery of Knowledge by Intrinsically Motivated Systems , 2015, ACALCI.

[76]  Shalabh Bhatnagar,et al.  Incremental Natural Actor-Critic Algorithms , 2007, NIPS.

[77]  A. Cangelosi,et al.  Developmental Robotics: From Babies to Robots , 2015 .

[78]  Giulio Sandini,et al.  Developmental Perception of the Self and Action , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[79]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[80]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[81]  G. Edelman,et al.  Synthetic neural modeling applied to a real-world artifact. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[82]  B Fritzke,et al.  A growing neural gas network learns topologies. G. Tesauro, DS Touretzky, and TK Leen, editors , 1995, NIPS 1995.

[83]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[84]  Francesco Mannella,et al.  The "Mechatronic Board": A Tool to Study Intrinsic Motivations in Humans, Monkeys, and Humanoid Robots , 2013, Intrinsically Motivated Learning in Natural and Artificial Systems.

[85]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[86]  Wulfram Gerstner,et al.  Mathematical formulations of Hebbian learning , 2002, Biological Cybernetics.

[87]  Marijn F. Stollenga,et al.  Continual curiosity-driven skill acquisition from high-dimensional video inputs for humanoid robots , 2017, Artif. Intell..

[88]  Pierre-Yves Oudeyer,et al.  Self-organization of early vocal development in infants and machines: the role of intrinsic motivation , 2014, Front. Psychol..

[89]  David Vernon,et al.  A Roadmap for Cognitive Development in Humanoid Robots , 2011, Cognitive Systems Monographs.

[90]  David Filliat,et al.  Exploration strategies for incremental learning of object-based visual saliency , 2015, 2015 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob).