Computational, Neuroscientific, and Lifespan Perspectives on the Exploration-Exploitation Dilemma

Computational, Neuroscientific, and Lifespan Perspectives on the Exploration-Exploitation Dilemma A. Ross Otto 1 (Moderator) , Bradley W. Knox 2 , and Bradley C. Love 1 Department of Psychology, 2 Department of Computer Science, University of Texas at Austin Sam Gershman and Yael Niv Department of Psychology, Princeton University Darrell A. Worthy 1 and W. Todd Maddox 2 Department of Psychology, Texas AM computational modeling; cogni- tive neuroscience, information search Consider the following real-life decisions that we make: deciding which route to take home to minimize time spent traveling, choosing amongst a set of known restaurants or a new restaurant when dining out, deciding between reading a new book by a consistently good author versus an author whose books vary widely in quality. All of these decisions involve balancing the conflicting demands of exploiting pre- vious knowledge in order maximize payoffs versus explor- ing less-known options in order to gain information about the currently optimal course of action. Indeed, successfully balancing these competing demands is a non-trivial problem of interest to artificial intelligence and neural Reinforcement Learning (RL) research communities alike (Cohen, McClure, & Yu, 2007; Daw et al., 2006; Sutton & Barto, 1998). There are adverse consequences for failing to properly balance these demands in the above examples: solely making exploita- tive choices entails the possibility of ignorance about better courses of action, while exploring too frequently incurs large opportunity costs. The goal of the proposed symposium is to bring together researchers from a variety of perspectives who are working to better understand the psychological and neurobiological mechanisms underlying exploratory choice. In recent years, novel computational modeling approaches have been developed and applied to understanding how hu- mans incorporate the demands of information gathering into their patterns of choice. These modeling techniques have yielded insight not only in describing human choice behav- ior, but also in understanding the neurobiological and physio- logical correlates of exploratory decision-making in humans (Daw et al., 2006; Jepma & Niewenhuis, in press). The re- searchers who have agreed to participate in this symposium are all applying computational models to better understand the psychological and neurobiological mechanisms under- pinning peoples negotiation of the exploration-exploitation tradeoff. The modeling approaches taken by these speakers are indeed diverse, ranging from uncovering hidden variables underlying decision-makers’ choices in order to unpack neu- robiological and physiological measurements to describing aging-related changes in exploratory choice behavior. The proposed symposium will provide a forum for highlighting recent advances in applications of computational modeling to human exploratory choice. The speakers who have agreed to participate in this symposium–while each performing research that elucidates psychological and neurobiological mechanisms underlying exploratory decision-making—offer different perspectives on the issue. The research described includes 1) aging work examining lifespan changes in exploratory decision-making, elucidating the underlying neurobiology of these types of choices (Worthy & Maddox), 2) a Bayesian account of effect of novelty–when humans are presented with new, potentially rewarding options–on exploratory choice, and how these nov- elty signals are represented in the brain in order to compute values and guide choices (Gershman & Niv), 3) how individu- als incorporate uncertainty and information search costs when planning courses of action in situations with sequential de- pendencies between choices and outcomes (Hotaling, Buse- meyer, & Shiffrin), and 4) how internally calculated uncer- tainty about the environment directs exploratory choice and manifests itself physiologically over the course of decision- making (Otto, Knox, & Love). In addition to proposing an- swers to a diverse set of important psychological and neuro- scientific questions, the lines of research described by these speakers rely upon laboratory tasks with monetary incentives that, each in their own way, incorporate ecologically interest- ing choice and reward dynamics. Belief-directed Exploration in Human Decision-Makers: Behavioral and Physiological Evidence A Ross Otto, W. Bradley Knox, & Bradley C. Love Decision-making in uncertain environments poses a con- flict between the goals of exploiting past knowledge in or- der to maximize rewards and exploring less-known options in order to gather information. However, the descriptive mod- eling framework utilized in previous studies of exploratory choice behavior characterizes exploration as the result of choices, rather than a process reflecting beliefs and/or un-