Improving the Reliability of Computational Analyses: Model-Based Planning and Its Relationship With Compulsivity.

BACKGROUND Computational models show great promise in mapping latent decision-making processes onto dissociable neural substrates and clinical phenotypes. One prominent example in reinforcement learning is model-based planning, which specifically relates to transdiagnostic compulsivity. However, the reliability of computational model-derived measures such as model-based planning is unclear. Establishing reliability is necessary to ensure that such models measure stable, traitlike processes, as assumed in computational psychiatry. Although analysis approaches affect validity of reinforcement learning models and reliability of other task-based measures, their effect on reliability of reinforcement learning models of empirical data has not been systematically studied. METHODS We first assessed within- and across-session reliability and effects of analysis approaches (model estimation, parameterization, and data cleaning) of measures of model-based planning in patients with compulsive disorders (n = 38). The analysis approaches affecting test-retest reliability were tested in 3 large generalization samples (healthy participants: n = 541 and 111; people with a range of compulsivity: n = 1413). RESULTS Analysis approaches greatly influenced reliability: reliability of model-based planning measures ranged from 0 (no concordance) to above 0.9 (acceptable for clinical applications). The largest influence on reliability was whether model-estimation approaches were robust and accounted for the hierarchical structure of estimated parameters. Improvements in reliability generalized to other datasets and greatly reduced the sample size needed to find a relationship between model-based planning and compulsivity in an independent dataset. CONCLUSIONS These results indicate that computational psychiatry measures such as model-based planning can reliably measure latent decision-making processes, but when doing so must assess the ability of methods to estimate complex models from limited data.

[1]  Thorsten Pachur,et al.  Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice , 2015, Psychonomic bulletin & review.

[2]  Martin P Paulus,et al.  A Roadmap for the Development of Applied Computational Psychiatry. , 2016, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[3]  M. Botvinick,et al.  Reduced model-based decision-making in schizophrenia. , 2016, Journal of abnormal psychology.

[4]  Wouter Kool,et al.  Incentives Boost Model-Based Control Across a Range of Severity on Several Psychiatric Constructs , 2019, Biological Psychiatry.

[5]  E. Wagenmakers,et al.  Using Bayesian regression to test hypotheses about relationships between parameters and covariates in cognitive models , 2017, Behavior research methods.

[6]  Miriam Sebold,et al.  When Habits Are Dangerous: Alcohol Expectancies and Habitual Decision Making Predict Relapse in Alcohol Dependence , 2017, Biological Psychiatry.

[7]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[8]  Michael Moutoussis,et al.  Hypotheses About the Relationship of Cognition With Psychopathology Should be Tested by Embedding Them Into Empirical Priors , 2018, Front. Psychol..

[9]  Alice Y. Chiang,et al.  Working-memory capacity protects model-based learning from stress , 2013, Proceedings of the National Academy of Sciences.

[10]  A. R. Otto,et al.  Model-based learning and individual differences in depression: The moderating role of stress. , 2018, Behaviour research and therapy.

[11]  Lionel Rigoux,et al.  VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data , 2014, PLoS Comput. Biol..

[12]  E. Koechlin,et al.  The Importance of Falsification in Computational Cognitive Modeling , 2017, Trends in Cognitive Sciences.

[13]  S. Gershman.  Empirical priors for reinforcement learning models , 2016 .

[14]  T. Rodebaugh,et al.  Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. , 2016, Journal of abnormal psychology.

[15]  Christopher K. Kovach,et al.  Anterior Prefrontal Cortex Contributes to Action Selection through Tracking of Recent Reward Trends , 2012, The Journal of Neuroscience.

[16]  A. Reiter,et al.  Model-Based Control in Dimensional Psychiatry , 2017, Biological Psychiatry.

[17]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[18]  Joshua W. Brown,et al.  A Model-Based fMRI Analysis with Hierarchical Bayesian Parameter Estimation , 2011 .

[19]  N. Daw,et al.  Dopamine selectively remediates 'model-based' reward learning: a computational approach. , 2016, Brain : a journal of neurology.

[20]  Greg J Siegle,et al.  Empirical recommendations for improving the stability of the dot-probe task in clinical research. , 2015, Psychological assessment.

[21]  Hideki Ohira,et al.  Biases in estimating the balance between model-free and model-based learning systems due to model misspecification , 2019, Journal of Mathematical Psychology.

[22]  P. Dayan,et al.  States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning , 2010, Neuron.

[23]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[24]  N. Daw,et al.  Variability in Dopamine Genes Dissociates Model-Based and Model-Free Reinforcement Learning , 2016, The Journal of Neuroscience.

[25]  Annchen R. Knodt,et al.  The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences , 2017, Behavior Research Methods.

[26]  Raymond J. Dolan,et al.  Go and no-go learning in reward and punishment: Interactions between affect and effect , 2012, NeuroImage.

[27]  C. Gillan,et al.  Does cognitive-behavioral therapy affect goal-directed planning in obsessive-compulsive disorder? , 2019, Psychiatry Research.

[28]  Michael Moutoussis,et al.  Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling , 2019, PLoS Comput. Biol..

[29]  Patrick G. Bissett,et al.  Large-scale analysis of test–retest reliabilities of self-regulation measures , 2019, Proceedings of the National Academy of Sciences.

[30]  Andrew Gelman,et al.  Measurement error and the replication crisis , 2017, Science.

[31]  M. Botvinick,et al.  The successor representation in human reinforcement learning , 2016, Nature Human Behaviour.

[32]  P. Glimcher,et al.  JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR 2005, 84, 555–579 NUMBER 3(NOVEMBER) DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS , 2022 .

[33]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[34]  M. S. Spektor,et al.  The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making , 2018, Psychonomic bulletin & review.

[35]  Christina L. Boisseau,et al.  Comparison of the Association Between Goal-Directed Planning and Self-reported Compulsivity vs Obsessive-Compulsive Disorder Diagnosis , 2019, JAMA psychiatry.

[36]  Marcelo G Mattar,et al.  Prioritized memory access explains planning and hippocampal replay , 2017, Nature Neuroscience.

[37]  Michael Moutoussis,et al.  Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood , 2018, PLoS Comput. Biol..

[38]  Simone Kühn,et al.  Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED) , 2018, eLife.

[39]  N. Daw,et al.  Model-based learning protects against forming habits , 2015, Cognitive, Affective, & Behavioral Neuroscience.

[40]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[41]  N. Daw,et al.  Characterizing a psychiatric symptom dimension related to deficits in goal-directed control , 2016, eLife.

[42]  Brooks King-Casas,et al.  Valuation in major depression is intact and stable in a non-learning environment , 2016, Scientific Reports.

[43]  Catherine A. Hartley,et al.  From Creatures of Habit to Goal-Directed Learners , 2016, Psychological science.

[44]  P. Dayan,et al.  Model-based influences on humans’ choices and striatal prediction errors , 2011, Neuron.

[45]  E. Wagenmakers,et al.  Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task , 2010 .

[46]  Jonathon R. Howlett,et al.  The effect of single-dose methylphenidate on the rate of error-driven learning in healthy males: a randomized controlled trial , 2017, Psychopharmacology.

[47]  Rebecca B. Price,et al.  Computational Modeling Applied to the Dot-Probe Task Yields Improved Reliability and Mechanistic Insights , 2019, Biological Psychiatry.