What are intrinsic motivations? A biological perspective

The concept of “intrinsic motivation”, initially proposed and developed within psychology, is gaining an increasing attention within cognitive sciences for its potential to produce open-ended learning machines and robots. However, a clear definition of the phenomenon is not yet available. This theoretical paper aims to clarify what intrinsic motivations are from a biological perspective. To this purpose, it first shows how intrinsic motivations can be defined contrasting them to extrinsic motivations from an evolutionary perspective: whereas extrinsic motivations guide learning of behaviours that directly increase fitness, intrinsic motivations drive the acquisition of knowledge and skills that contribute to produce behaviours that increase fitness only in a later stage. Given this difference, extrinsic motivations generate learning signals on the basis of events involving body homeostatic regulations, whereas intrinsic motivations generate learning signals based on events taking place within the brain itself. These ideas are supported by presenting some examples of biological mechanisms underlying the two types of motivations. The paper closes by linking the theory to the current major computational views on intrinsic motivations and by listing the main open issues of the field.

[1]  James L Olds,et al.  Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. , 1954, Journal of comparative and physiological psychology.

[2]  Marco Mirolli,et al.  Evolving Childhood's Length and Learning Parameters in an Intrinsically Motivated Reinforcement Learning Robot , 2007 .

[3]  G. B. Kish.  Learning when the onset of illumination is used as reinforcing stimulus. , 1955, Journal of comparative and physiological psychology.

[4]  Kenji Matsumoto,et al.  Neural basis of the undermining effect of monetary reward on intrinsic motivation , 2010, Proceedings of the National Academy of Sciences.

[5]  B. Babkin.  Conditioned Reflexes; an Investigation of the Physiological Activity of the Cerebral Cortex. , 1929 .

[6]  P. Milner,et al.  Brain-stimulation reward: a review. , 1991, Canadian journal of psychology.

[7]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[8]  S. Paradiso,et al.  Book Review: Affective Neuroscience: The Foundations of Human and Animal Emotions , 2000 .

[9]  K. Lorenz,et al.  King Solomon's Ring , 1949 .

[10]  H. Birx,et al.  The Mismeasure of Man , 1981 .

[11]  Andrew G. Barto,et al.  Competence progress intrinsic motivation , 2010, 2010 IEEE 9th International Conference on Development and Learning.

[12]  Pierre-Yves Oudeyer,et al.  In Search of the Neural Circuits of Intrinsic Motivation , 2007, Front. Neurosci..

[13]  B. Skinner,et al.  Principles of Behavior , 1944 .

[14]  William Rowan,et al.  The Study of Instinct , 1953 .

[15]  Hugo Vieira Neto,et al.  Visual novelty detection with automatic scale selection , 2007, Robotics Auton. Syst..

[16]  J. Rothwell.  Principles of Neural Science , 1982 .

[17]  Domenico Parisi,et al.  A Bioinspired Hierarchical Reinforcement Learning Architecture for Modeling Learning of Multiple Skills with Continuous States and Actions , 2010, EpiRob.

[18]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[19]  E. Stricker,et al.  Neurobiology of Food and Fluid Intake , 2011, Handbook of Behavioral Neurobiology.

[20]  Jürgen Schmidhuber,et al.  Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010) , 2010, IEEE Transactions on Autonomous Mental Development.

[21]  Peter Dayan,et al.  Expected and Unexpected Uncertainty: ACh and NE in the Neocortex , 2002, NIPS.

[22]  Stefano Nolfi,et al.  Econets: Neural networks that learn in an environment , 1990 .

[23]  L. D. Reid,et al.  Endogenous opioid peptides and regulation of drinking and feeding. , 1985, The American journal of clinical nutrition.

[24]  Irving Kupfermann,et al.  Neural control of feeding , 1994, Current Opinion in Neurobiology.

[25]  C Gianoulakis,et al.  Biosynthesis of beta-endorphin from beta-lipotropin and a larger molecular weight precursor in rat pars intermedia. , 1978, Proceedings of the National Academy of Sciences of the United States of America.

[26]  Jürgen Schmidhuber,et al.  Curious model-building control systems , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[27]  F. Weiss,et al.  The dopamine hypothesis of reward: past and current status , 1999, Trends in Neurosciences.

[28]  B. Lowell,et al.  Identifying hypothalamic pathways controlling food intake, body weight, and glucose homeostasis , 2005, The Journal of comparative neurology.

[29]  D. Berlyne.  Curiosity and exploration. , 1966, Science.

[30]  Richard L. Lewis,et al.  Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective , 2010, IEEE Transactions on Autonomous Mental Development.

[31]  B. Roche,et al.  The Behavior of Organisms? , 1997 .

[32]  J D Corbit,et al.  Voluntary control of hypothalamic temperature. , 1973, Journal of comparative and physiological psychology.

[33]  W. Schultz.  Getting Formal with Dopamine and Reward , 2002, Neuron.

[34]  T. Johnston.  Selective costs and benefits of in the evolution of learning , 1996 .

[35]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[36]  Domenico Parisi,et al.  Internal robotics , 2004, Connect. Sci..

[38]  Pierre-Yves Oudeyer,et al.  What is Intrinsic Motivation? A Typology of Computational Approaches , 2007, Frontiers Neurorobotics.

[39]  D. S. Zahm,et al.  Glutamatergic Afferents of the Ventral Tegmental Area in the Rat , 2007, The Journal of Neuroscience.

[40]  J. Lisman,et al.  The Hippocampal-VTA Loop: Controlling the Entry of Information into Long-Term Memory , 2005, Neuron.

[41]  D. Kumaran,et al.  Which computational mechanisms operate in the hippocampus during novelty detection? , 2007, Hippocampus.

[42]  Philip N. Lehner,et al.  Handbook of ethological methods , 1979 .

[43]  J. Boulant,et al.  Role of the preoptic-anterior hypothalamus in thermoregulation and fever. , 2000, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[44]  S. Sara.  The locus coeruleus and noradrenergic modulation of cognition , 2009, Nature Reviews Neuroscience.

[45]  I. Pavlov.  Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex , 1929 .

[46]  Nuttapong Chentanez,et al.  Intrinsically Motivated Reinforcement Learning , 2004, NIPS.

[47]  R. W. White.  Motivation reconsidered: the concept of competence. , 1959, Psychological review.

[48]  S. Sara,et al.  Locus coeruleus-evoked responses in behaving rats: A clue to the role of noradrenaline in memory , 1994, Brain Research Bulletin.

[49]  N. Tinbergen.  On aims and methods of Ethology , 2010 .

[50]  Kyle S. Smith,et al.  Hedonic Hot Spots in the Brain , 2006, The Neuroscientist : a review journal bringing neurobiology, neurology and psychiatry.

[51]  Francesco Mannella,et al.  The roles of the amygdala in the affective regulation of body, brain, and behaviour , 2010, Connect. Sci..

[52]  Harlow Hf.  Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. , 1950 .

[53]  Marco Mirolli,et al.  Biological Cumulative Learning through Intrinsic Motivations: A Simulated Robotic Study on the Development of Visually-Guided Reaching , 2010, EpiRob.

[54]  Christian Balkenius,et al.  Proceedings of the Seventh International Conference on Epigenetic Robotics , 2007 .