Unstable Dynamics of Adaptation in unknown Environment due to Novelty seeking

Learning and adaptation play great role in emergent socio-economic phenomena. Complex dynamics has been previously found in the systems of multiple learning agents interacting via a simple game. Meanwhile, the single agent adaptation is considered trivially stable. We advocate the idea that adopting a more complex model of the individual behavior may result in a more diverse spectrum of macro-level behaviors. We develop an adaptation model based on the reinforcement learning framework extended by an additional processing channel. We scrutiny the dynamics of the single agent adapting to the unknown environment; the agent is biased by novelty seeking, the intrinsic inclination for exploration. We demonstrate that the behavior of the novelty-seeking agent may be inherently unstable. One of the surprising results is that under certain conditions the increase of the novelty-seeking level may cause the agent to switch from the non-rational to the strictly rational behavior. Our results give evidence to the hypothesis that the intrinsic motives of agents should be paid no less attention than the extrinsic ones in the models of complex socio-economic systems.

[1]  I. Erev,et al.  Small feedback‐based decisions and their limited correspondence to description‐based decisions , 2003 .

[2]  Pierre-Yves Oudeyer,et al.  What is Intrinsic Motivation? A Typology of Computational Approaches , 2007, Frontiers Neurorobotics.

[3]  David S. Leslie,et al.  Individual Q-Learning in Normal Form Games , 2005, SIAM J. Control. Optim..

[4]  Tobias Galla,et al.  Intrinsic noise in game dynamical learning. , 2009, Physical review letters.

[5]  Jonathan Evans Dual-processing accounts of reasoning, judgment, and social cognition. , 2008, Annual review of psychology.

[6]  Eric J. Johnson,et al.  Mindful judgment and decision making. , 2009, Annual review of psychology.

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  Cloninger Cr Discussions arising from: Cloninger, CR. A. unified biosocial theory of personality and its role in the development of anxiety states. , 1987, Psychiatric developments.

[9]  Xin Wang,et al.  Individual Differences in EWA Learning with Partial Payoff Information , 2008 .

[10]  Tobias Galla,et al.  Cycles of cooperation and defection in imperfect learning , 2011, ArXiv.

[11]  Ihor Lubashevsky,et al.  Scale-free memory model for multiagent reinforcement learning. Mean field approximation and rock-paper-scissors dynamics , 2009, 0911.2406.

[12]  Edward L. Deci,et al.  Intrinsic Motivation and Self-Determination in Human Behavior , 1975, Perspectives in Social Psychology.

[13]  Ihor Lubashevsky,et al.  Extended phase space description of human-controlled systems dynamics , 2012, 1212.2717.

[14]  John L. Crompton,et al.  Measuring novelty seeking in tourism. , 1992 .

[15]  W. Arthur Inductive Reasoning and Bounded Rationality , 1994 .

[16]  Aram Galstyan,et al.  Dynamics of Boltzmann Q learning in two-player two-action games. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[17]  Tilman Börgers,et al.  Learning Through Reinforcement and Replicator Dynamics , 1997 .

[18]  K. Mukherjee A dual system model of preferences under risk. , 2010, Psychological review.

[19]  A. Roth,et al.  Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria , 1998 .

[20]  J. Nadal,et al.  Manifesto of computational social science , 2012 .

[21]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[22]  M. Macy,et al.  Learning dynamics in social dilemmas , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[23]  T. Ordoña,et al.  Feeling Good: The Science of Well-Being , 2005 .

[24]  Keith Frankish,et al.  Dual-Process and Dual-System Theories of Reasoning , 2010 .

[25]  Jean-Louis Deneubourg,et al.  Self-organization or individual complexity: a false dilemma or a true complementarity? , 1999 .

[26]  Joan Y. Chiao,et al.  Genetic Determinants of Financial Risk Taking , 2009, PloS one.

[27]  A. Tversky,et al.  Judgment under Uncertainty , 1982 .

[28]  C. R. Cloninger,et al.  A psychobiological model of temperament and character. , 1993, Archives of general psychiatry.

[29]  Jennifer Trueblood,et al.  A Dynamic Dual-Process Model of Decision-making Under Uncertainty , 2013, CogSci.

[30]  J. Crutchfield,et al.  Coupled replicator equations for the dynamics of learning in multiagent systems. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  Timothy E. J. Behrens,et al.  Choice, uncertainty and value in prefrontal and cingulate cortex , 2008, Nature Neuroscience.

[32]  E. Deci,et al.  Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. , 2000, The American psychologist.

[33]  P. Dayan,et al.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control , 2005, Nature Neuroscience.

[34]  A Thorne,et al.  Personal Memory Telling and Personality Development , 2000, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[35]  Eizo Akiyama,et al.  Chaos in learning a simple two-person game , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[36]  F. Roces Individual Complexity and Self-Organization in Foraging by Leaf-Cutting Ants , 2002, The Biological Bulletin.

[37]  E. Deci,et al.  Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. , 2000, Contemporary educational psychology.

[38]  Angela J. Yu,et al.  Emotion and decision-making: affect-driven belief systems in anxiety and depression , 2012, Trends in Cognitive Sciences.

[39]  D. Fudenberg,et al.  A Dual Self Model of Impulse Control , 2004, The American economic review.

[40]  James P. Crutchfield,et al.  Stability and diversity in collective adaptation , 2004, nlin/0408039.

[41]  D. Fudenberg,et al.  The Theory of Learning in Games , 1998 .