The Outcome-Representation Learning Model: A Novel Reinforcement Learning Model of the Iowa Gambling Task

The Iowa Gambling Task (IGT) is widely used to study decision-making within healthy and psychiatric populations. However, the complexity of the IGT makes it difficult to attribute variation in performance to specific cognitive processes. Several cognitive models have been proposed for the IGT in an effort to address this problem, but currently no single model shows optimal performance for both short- and long-term prediction accuracy and parameter recovery. Here, we propose the Outcome-Representation Learning (ORL) model, a novel model that provides the best compromise between competing models. We test the performance of the ORL model on 393 subjects' data collected across multiple research sites, and we show that the ORL reveals distinct patterns of decision-making in substance-using populations. Our work highlights the importance of using multiple model comparison metrics to make valid inference with cognitive models and sheds light on learning mechanisms that play a role in underweighting of rare events.

[1]  Jerome R. Busemeyer,et al.  Computational Modeling Reveals Distinct Effects of HIV and History of Drug Use on Decision-Making Processes in Women , 2013, PloS one.

[2]  Kevin McCabe,et al.  Neural signature of fictive learning signals in a sequential investment task , 2007, Proceedings of the National Academy of Sciences.

[3]  Eric-Jan Wagenmakers,et al.  A Comparison of Reinforcement Learning Models for the Iowa Gambling Task Using Parameter Space Partitioning , 2013, J. Probl. Solving.

[4]  James A. Waltz,et al.  Interactions Among Working Memory, Reinforcement Learning, and Effort in Value-Based Choice: A New Paradigm and Selective Deficits in Schizophrenia , 2017, Biological Psychiatry.

[5]  Timothy J. Pleskac,et al.  Modeling behavior in a clinically diagnostic sequential risk-taking task. , 2005, Psychological review.

[6]  Jun Lu,et al.  An introduction to Bayesian hierarchical models with an application in the theory of signal detection , 2005, Psychonomic bulletin & review.

[7]  Yao-Chu Chiu,et al.  Is deck C an advantageous deck in the Iowa Gambling Task? , 2007, Behavioral and Brain Functions.

[8]  Darrell A. Worthy,et al.  Heterogeneity of strategy use in the Iowa gambling task: A comparison of win-stay/lose-shift and reinforcement learning models , 2013, Psychonomic bulletin & review.

[9]  I. Erev,et al.  On adaptation, maximization, and reinforcement learning among cognitive strategies. , 2005, Psychological review.

[10]  M. Frank,et al.  Instructional control of reinforcement learning: A behavioral and neurocomputational investigation , 2009, Brain Research.

[11]  T. Salthouse,et al.  Performance on the Iowa Gambling Task: From 5 to 89 years of age. , 2014, Journal of experimental psychology. General.

[12]  Michael J. Frank,et al.  By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism , 2004, Science.

[13]  E. Wagenmakers,et al.  Absolute performance of reinforcement-learning models for the Iowa Gambling Task , 2014 .

[14]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[15]  J. Kruschke,et al.  Using cognitive science methods to assess the role of social information processing in sexually coercive behavior. , 2001, Psychological assessment.

[16]  Ralph Neuneier,et al.  Risk-Sensitive Reinforcement Learning , 1998, Machine Learning.

[17]  W. Batchelder Multinomial processing tree models and psychological assessment. , 1998 .

[18]  Zhong-Lin Lu,et al.  Neural correlates of risk prediction error during reinforcement learning in humans , 2009, NeuroImage.

[19]  Jerome R. Busemeyer,et al.  Using Cognitive Models to Map Relations Between Neuropsychological Disorders and Human Decision-Making Deficits , 2005, Psychological science.

[20]  Darrell A. Worthy,et al.  To not settle for small losses: evidence for an ecological aspiration level of zero in dynamic decision-making , 2017, Psychonomic bulletin & review.

[21]  Michael J. Frank,et al.  Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning , 2007, Proceedings of the National Academy of Sciences.

[22]  Carlo Contoreggi,et al.  Drug abusers show impaired performance in a laboratory test of decision making , 2000, Neuropsychologia.

[23]  P. Laurienti,et al.  Long-term heavy marijuana users make costly decisions on a gambling task. , 2004, Drug and alcohol dependence.

[24]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[25]  Aki Vehtari,et al.  Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC , 2015, Statistics and Computing.

[26]  J. Pearce,et al.  A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. , 1980 .

[27]  E. Wagenmakers,et al.  Diffusion versus linear ballistic accumulation: different models but the same conclusions about psychological processes? , 2010, Psychonomic bulletin & review.

[28]  Woo-Young Ahn,et al.  Challenges and promises for translating computational tools into clinical practice , 2016, Current Opinion in Behavioral Sciences.

[29]  E. Siemers,et al.  Risky decision making in Huntington's disease , 2001, Journal of the International Neuropsychological Society.

[30]  Woojae Kim,et al.  Cognitive Mechanisms Underlying Risky Decision-Making in Chronic Cannabis Users. , 2010, Journal of mathematical psychology.

[31]  M. Farah,et al.  Different underlying impairments in decision-making following ventromedial and dorsolateral frontal lobe damage in humans. , 2004, Cerebral cortex.

[32]  A. Damasio,et al.  Insensitivity to future consequences following damage to human prefrontal cortex , 1994, Cognition.

[33]  J. Busemeyer,et al.  Older adults as adaptive decision makers: evidence from the Iowa Gambling Task. , 2005, Psychology and aging.

[34]  J. Hsieh,et al.  Immediate gain is long-term loss: Are there foresighted decision makers in the Iowa Gambling Task? , 2008, Behavioral and Brain Functions.

[35]  A. Roth,et al.  Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria , 1998 .

[36]  A. Simmons,et al.  Emotional decision-making and its dissociable components in schizophrenia and schizoaffective disorder: A behavioural and MRI investigation , 2008, Neuropsychologia.

[37]  Steven W Anderson,et al.  Decision-making deficits, linked to a dysfunctional ventromedial prefrontal cortex, revealed in alcohol and stimulant abusers , 2001, Neuropsychologia.

[38]  Scott D. Lane,et al.  Relationship between impulsivity and decision making in cocaine dependence , 2010, Psychiatry Research.

[39]  Woo-Young Ahn,et al.  Machine-learning identifies substance-specific behavioral markers for opiate and stimulant dependence. , 2016, Drug and alcohol dependence.

[40]  Kaileigh A. Byrne,et al.  Decomposing the roles of perseveration and expected value representation in models of the Iowa gambling task , 2013, Front. Psychol..

[41]  Eldad Yechiam,et al.  Evaluating the reliance on past choices in adaptive learning models , 2007 .

[42]  John K. Kruschke,et al.  Decision-making in stimulant and opiate addicts in protracted abstinence: evidence from computational modeling with pure users , 2014, Front. Psychol..

[43]  N. Roese,et al.  What We Regret Most... and Why , 2005, Personality & social psychology bulletin.

[44]  L. M. Lieving,et al.  Acute Marijuana Effects on Human Risk Taking , 2005, Neuropsychopharmacology.

[45]  Michael D. Lee,et al.  A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods , 2008, Cogn. Sci..

[46]  R. Hertwig,et al.  Decisions from Experience and the Effect of Rare Events in Risky Choice , 2004, Psychological science.

[47]  Sergejs Fomins,et al.  Color-discrimination threshold determination using pseudoisochromatic test plates , 2014, Front. Psychol..

[48]  Aki Vehtari,et al.  Understanding predictive information criteria for Bayesian models , 2013, Statistics and Computing.

[49]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[50]  R. Neufeld,et al.  Application of stochastic modeling to the assessment of group and individual differences in cognitive functioning. , 2002, Psychological Assessment.

[51]  Eric-Jan Wagenmakers,et al.  An EZ-diffusion model for response time and accuracy , 2007, Psychonomic bulletin & review.

[52]  Sylvia M. L. Cox,et al.  Striatal D1 and D2 signaling differentially predict learning from positive and negative outcomes , 2015, NeuroImage.

[53]  T. Maiti,et al.  Regional fertility data analysis: A small area Bayesian approach , 2013 .

[54]  E. Yechiam,et al.  Loss aversion, diminishing sensitivity, and the effect of experience on repeated decisions† , 2008 .

[55]  J. Busemeyer,et al.  Computational modeling for addiction medicine: From cognitive models to clinical applications. , 2016, Progress in brain research.

[56]  S. Lane,et al.  Marijuana Effects on Sensitivity to Reinforcement in Humans , 2002, Neuropsychopharmacology.

[57]  Merrill Hiscock,et al.  The Simulated Gambling Paradigm Applied to Young Adults: An Examination of University Students' Performance , 2006, Applied neuropsychology.

[58]  R. Ratcliff,et al.  Explicitly modeling the effects of aging on response time , 2000, Psychonomic bulletin & review.

[59]  J. Gläscher,et al.  Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. , 2009, Cerebral cortex.

[60]  R. Hertwig,et al.  The description–experience gap in risky choice , 2009, Trends in Cognitive Sciences.

[61]  Samuel J Gershman,et al.  Do learning rates adapt to the distribution of rewards? , 2015, Psychonomic bulletin & review.

[62]  I. Erev,et al.  Small feedback‐based decisions and their limited correspondence to description‐based decisions , 2003 .

[63]  Keith H. Nuechterlein,et al.  Schizophrenia patients demonstrate a distinctive pattern of decision-making impairment on the Iowa Gambling Task , 2005, Schizophrenia Research.

[64]  E. Koechlin,et al.  The Importance of Falsification in Computational Cognitive Modeling , 2017, Trends in Cognitive Sciences.

[65]  J. O'Doherty,et al.  The Role of the Ventromedial Prefrontal Cortex in Abstract State-Based Inference during Decision Making in Humans , 2006, The Journal of Neuroscience.

[66]  M. Lee How cognitive modeling can benefit from hierarchical Bayesian models. , 2011 .

[67]  Jerome R. Busemeyer,et al.  Comparison of Decision Learning Models Using the Generalization Criterion Method , 2008, Cogn. Sci..

[68]  James T. Townsend,et al.  Foundations of psychological assessment: Implications for cognitive assessment in clinical science. , 1998 .

[69]  Joshua W. Brown,et al.  A Model-Based fMRI Analysis with Hierarchical Bayesian Parameter Estimation , 2011 .

[70]  Hanna Damasio,et al.  Decision-making and addiction (part I): impaired activation of somatic states in substance dependent individuals when pondering decisions with negative future consequences , 2002, Neuropsychologia.

[71]  J. Busemeyer,et al.  A contribution of cognitive decision models to clinical assessment: decomposing performance on the Bechara gambling task. , 2002, Psychological assessment.

[72]  E. Wagenmakers,et al.  Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task , 2010 .