Influenca: a gamified assessment of value-based decision-making for longitudinal studies

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. Here, we developed an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) for repeated assessment over weeks. In this task, players have to identify the most effective medication by selecting the best option after integrating offered points with changing probabilities (according to random Gaussian walks). Participants can complete up to 31 levels with 150 trials each. To encourage replay on their preferred device, in-game screens provide feedback on the progress. Using an initial validation sample of 127 players (2904 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show low to medium intra-class correlations (ICC: 0.22-0.52), indicating substantial within- and between-subject variance. Notably, state items showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

[1]  C. Gillan,et al.  Smartphones and the Neuroscience of Mental Health. , 2021, Annual review of neuroscience.

[2]  J. Steinglass,et al.  Reward Learning Through the Lens of RDoC: a Review of Theory, Assessment, and Empirical Findings in the Eating Disorders , 2021, Current Psychiatry Reports.

[3]  J. Ragland,et al.  Reliability and Replicability of Implicit and Explicit Reinforcement Learning Paradigms in People With Psychotic Disorders. , 2020, Schizophrenia bulletin.

[4]  R. Rutledge,et al.  Momentary subjective well-being depends on learning and not reward , 2020, eLife.

[5]  N. Kroemer,et al.  Beyond the average: The role of variable reward sensitivity in eating disorders , 2020, Physiology & Behavior.

[6]  M. Berking,et al.  Prospective associations of negative mood and emotion regulation in the occurrence of binge eating in binge eating disorder. , 2019, Journal of psychiatric research.

[7]  Juliane H. Fröhner,et al.  Addressing the reliability fallacy in fMRI: Similar group effects may arise from unreliable individual effects , 2019, NeuroImage.

[8]  R. Dolan,et al.  Older adults fail to form stable task representations during model-based reversal inference , 2019, Neurobiology of Aging.

[9]  Michael Moutoussis,et al.  Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood , 2018, PLoS Comput. Biol..

[10]  Patrick G. Bissett,et al.  Large-scale analysis of test–retest reliabilities of self-regulation measures , 2018, Proceedings of the National Academy of Sciences.

[11]  Kathryn E. Smith,et al.  Associations among eating disorder behaviors and eating disorder quality of life in adult women with anorexia nervosa , 2018, Psychiatry Research.

[12]  James C. Thompson,et al.  The impact of neural responses to food cues following stress on trajectories of negative and positive affect and binge eating in daily life. , 2018, Journal of psychiatric research.

[13]  Andrew Chiu,et al.  Prospect Theory , 2018, Encyclopedia of Wireless Networks.

[14]  J. Roiser,et al.  Neuroscience of apathy and anhedonia: a transdiagnostic approach , 2018, Nature Reviews Neuroscience.

[15]  Udo Boehm,et al.  On the importance of avoiding shortcuts in applying cognitive models to hierarchical data , 2018, Behavior Research Methods.

[16]  Melissa T. Buelow,et al.  Test–Retest Reliability of Common Behavioral Decision Making Tasks , 2018, Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists.

[17]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[18]  P. Dayan,et al.  Modeling Avoidance in Mood and Anxiety Disorders Using Reinforcement Learning , 2017, Biological Psychiatry.

[19]  Annchen R. Knodt,et al.  The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences , 2017, Behavior research methods.

[20]  Jenifer Z. Siegel,et al.  Association of Neural and Emotional Impacts of Reward Prediction Errors With Major Depression , 2017, JAMA psychiatry.

[21]  L. Deserno,et al.  Reversal learning strategy in adolescence is associated with prefrontal cortex activation , 2017, The European journal of neuroscience.

[22]  Nils B. Kroemer,et al.  Fuel not fun: Reinterpreting attenuated brain responses to reward in obesity , 2016, Physiology & Behavior.

[23]  T. Rodebaugh,et al.  Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias. , 2016, Journal of abnormal psychology.

[24]  R. Adolphs,et al.  Building a Science of Individual Differences from fMRI , 2016, Trends in Cognitive Sciences.

[25]  N. Daw,et al.  Characterizing a psychiatric symptom dimension related to deficits in goal-directed control , 2016, eLife.

[26]  Taiki Takahashi,et al.  Reinforcement learning in depression: A review of computational research , 2015, Neuroscience & Biobehavioral Reviews.

[27]  M. Jones-Gotman,et al.  Working memory and reward association learning impairments in obesity , 2014, Neuropsychologia.

[28]  J. Simons,et al.  Daily associations between anxiety and alcohol use: variation by sustained attention, set shifting, and gender. , 2014, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[29]  P. Dayan,et al.  A computational and neural model of momentary subjective well-being , 2014, Proceedings of the National Academy of Sciences.

[30]  Gerjo Kok,et al.  Monitoring Dietary Intake and Physical Activity Electronically: Feasibility, Usability, and Ecological Validity of a Mobile-Based Ecological Momentary Assessment Tool , 2013, Journal of medical Internet research.

[31]  P. Dayan,et al.  Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis , 2013, Biology of Mood & Anxiety Disorders.

[32]  Todd M. Gureckis,et al.  CUNY Academic , 2016 .

[33]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[34]  P. Dayan,et al.  Dopamine and performance in a reinforcement learning task: evidence from Parkinson's disease. , 2012 .

[35]  Anatol C. Kreitzer,et al.  Distinct roles for direct and indirect pathway striatal neurons in reinforcement , 2012, Nature Neuroscience.

[36]  Karl J. Friston,et al.  Computational psychiatry , 2012, Trends in Cognitive Sciences.

[37]  Michael D. Lee,et al.  Psychological models of human and optimal performance in bandit problems , 2011, Cognitive Systems Research.

[38]  T. Robbins,et al.  Decision Making, Affect, and Learning: Attention and Performance XXIII , 2011 .

[39]  Karl J. Friston,et al.  A Bayesian Foundation for Individual Learning Under Uncertainty , 2011, Front. Hum. Neurosci..

[40]  Nathaniel D. Daw,et al.  Trial-by-trial data analysis using computational models , 2011 .

[41]  D. Gilbert,et al.  A Wandering Mind Is an Unhappy Mind , 2010, Science.

[42]  J. Roiser,et al.  Reward and Punishment Processing in Depression , 2010, Biological Psychiatry.

[43]  S. Shiffman Ecological momentary assessment (EMA) in studies of substance use. , 2009, Psychological assessment.

[44]  T. Trull,et al.  Ecological momentary assessment of mood disorders and mood dysregulation. , 2009, Psychological assessment.

[45]  Rachel M. Calogero,et al.  Ecological momentary assessment of affect, stress, and binge-purge behaviors: day of week and time of day effects in the natural environment. , 2009, The International journal of eating disorders.

[46]  Daniel G. Dillon,et al.  Individual differences in reinforcement learning: Behavioral, electrophysiological, and neuroimaging correlates , 2008, NeuroImage.

[47]  Timothy E. J. Behrens,et al.  Learning the value of information in an uncertain world , 2007, Nature Neuroscience.

[48]  Angela J. Yu,et al.  Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[49]  Sabrina M. Tom,et al.  The Neural Basis of Loss Aversion in Decision-Making Under Risk , 2007, Science.

[50]  P. Dayan,et al.  Cortical substrates for exploratory decisions in humans , 2006, Nature.

[51]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[52]  M. Perrez,et al.  How to assess social regulation of stress and emotions in daily family life? A computer-assisted family self-monitoring system (FASEM-C) , 2000 .

[53]  A. Tversky,et al.  Advances in prospect theory: Cumulative representation of uncertainty , 1992 .

[54]  Richard Taylor Interpretation of the Correlation Coefficient: A Basic Review , 1990 .

[55]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[56]  Laura Weidinger,et al.  Test-retest reliability of canonical reinforcement learning models , 2019, 2019 Conference on Cognitive Computational Neuroscience.

[57]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[58]  Tyler B. Wray,et al.  Using Ecological Momentary Assessment (EMA) to Assess Situation-Level Predictors of Alcohol Use and Alcohol-Related Consequences , 2014, Alcohol research : current reviews.

[59]  R. Poldrack,et al.  Prospect Theory and the Brain , 2009 .

[60]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[61]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.