What is Intrinsic Motivation? A Typology of Computational Approaches

Intrinsic motivation, centrally involved in spontaneous exploration and curiosity, is a crucial concept in developmental psychology. It has been argued to be a crucial mechanism for open-ended cognitive development in humans, and as such has gathered a growing interest from developmental roboticists in the recent years. The goal of this paper is threefold. First, it provides a synthesis of the different approaches of intrinsic motivation in psychology. Second, by interpreting these approaches in a computational reinforcement learning framework, we argue that they are not operational and even sometimes inconsistent. Third, we set the ground for a systematic operational study of intrinsic motivation by presenting a formal typology of possible computational approaches. This typology is partly based on existing computational models, but also presents new ways of conceptualizing intrinsic motivation. We argue that this kind of computational typology might be useful for opening new avenues for research both in psychology and developmental robotics.

[1]  C. L. Hull.  Principles of behavior : an introduction to behavior theory , 1943 .

[2]  H. Harlow.  Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. , 1950, Journal of comparative and physiological psychology.

[3]  Harlow Hf.  Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. , 1950 .

[4]  K. Montgomery.  The role of the exploratory drive in learning. , 1954, Journal of comparative and physiological psychology.

[5]  W. N. Dember,et al.  Analysis of exploratory, manipulatory, and curiosity behaviors. , 1957, Psychological review.

[6]  L. Festinger,et al.  A Theory of Cognitive Dissonance , 2017 .

[7]  R. W. White.  Motivation reconsidered: the concept of competence. , 1959, Psychological review.

[8]  R. Decharms.  Personal causation : the internal affective determinants of behavior , 1968 .

[9]  J. Watson.  Smiling, cooing, and "the game." , 1972 .

[10]  J. Kagan.  Motives and development. , 1972, Journal of personality and social psychology.

[11]  V. Braitenberg.  Vehicles, Experiments in Synthetic Psychology , 1984 .

[12]  Edward L. Deci,et al.  Intrinsic Motivation and Self-Determination in Human Behavior , 1975, Perspectives in Social Psychology.

[13]  J. Crutchfield.  Information and Its Metric , 1990 .

[14]  Jürgen Schmidhuber,et al.  Curious model-building control systems , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[15]  D. McFarland,et al.  Intelligent behavior in animals and robots , 1993 .

[16]  Ronald C. Arkin,et al.  Ecological Robotics: A Schema-Theoretic Approach , 1996, Intelligent Robots.

[17]  Sebastian Thrun,et al.  Exploration in active learning , 1998 .

[18]  L. Gogate,et al.  Intersensory redundancy facilitates learning of arbitrary relations between vowel sounds and objects in seven-month-old infants. , 1998, Journal of experimental child psychology.

[19]  Javier R. Movellan,et al.  Audio Vision: Using Audio-Visual Synchrony to Locate Sounds , 1999, NIPS.

[20]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[21]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[22]  P. Rochat,et al.  Perceived self in infancy , 2000 .

[23]  Andrew McCallum,et al.  Toward Optimal Active Learning through Sampling Estimation of Error Reduction , 2001, ICML.

[24]  Ronald C. Arkin,et al.  Implementing Tolman's schematic sowbug: behavior-based robotics in the 1930's , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[25]  Xiao Huang,et al.  Novelty and Reinforcement Learning in the Value System of Developmental Robots , 2002 .

[26]  Cynthia Breazeal,et al.  Designing sociable robots , 2002 .

[27]  Pierre-Yves Oudeyer,et al.  Motivational principles for visual know-how development , 2003 .

[28]  R. Arkin.  Moving Up the Food Chain: Motivation and Emotion in Behavior-Based Robots , 2003 .

[29]  Christopher G. Prince,et al.  Taking Synchrony Seriously: A Perceptual-Level Model of Infant Synchrony Detection , 2004 .

[30]  Juyang Weng,et al.  Motivational System for Human-Robot Interaction , 2004, ECCV Workshop on HCI.

[31]  Nuttapong Chentanez,et al.  Intrinsically Motivated Learning of Hierarchical Collections of Skills , 2004 .

[32]  Douglas S. Blank,et al.  An Emergent Framework For Self-Motivation In Developmental Robotics , 2004 .

[33]  Juergen Konczak,et al.  Fourth International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems , 2004, Cognitive Systems Research.

[34]  Pierre-Yves Oudeyer,et al.  The Playground Experiment: Task-Independent Development of a Curious Robot , 2005 .

[35]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[36]  A. Barto,et al.  Intrinsic Motivation For Reinforcement Learning Systems , 2005 .

[37]  Andrew G. Barto,et al.  An Adaptive Robot Motivational System , 2006, SAB.

[38]  Pierre-Yves Oudeyer,et al.  Discovering communication , 2006, Connect. Sci..

[39]  Andrea Bonarini,et al.  Self-Development Framework for Reinforcement Learning Agents , 2006 .

[40]  Pierre-Yves Oudeyer,et al.  The progress drive hypothesis: an interpretation of early imitation , 2007 .

[41]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[42]  Pierre-Yves Oudeyer,et al.  In Search of the Neural Circuits of Intrinsic Motivation , 2007, Front. Neurosci..

[43]  Kathryn E. Merrick,et al.  Motivated Learning from Interesting Events: Adaptive, Multitask Learning Agents for Complex Environments , 2009, Adapt. Behav..