Manipulating and decoding subjective gaming experience during active gameplay: a multivariate, whole-brain analysis

ABSTRACT A large number of perceptual and cognitive processes are instantiated during active gameplay, culminating in what is termed the overall “gaming experience”, which encapsulates multiple, subjective dimensions of how one feels about the game. Although some research has been conducted into the neural mechanisms underlying the gaming experience, previous studies so far have relied on commercial games that provide little control over key aspects of gameplay and also have focused only on a few individual dimensions of the gaming experience. Here, we used a custom‐made, immersive driving car game in four different gameplay versions (baseline, obstacle increase, goal decrease, speed increase) to assess and modulate the subjective gameplay experience while participants underwent a fMRI scan. A multivariate correlation analysis of whole‐brain neural activity with behaviorally‐identified subjective gaming experience uncovered brain networks associated with different experiences, including higher‐level visual processing networks, the default network, and emotional areas. These regions were in addition able to decode the four different game conditions above chance. Our results for the first time describe the full range of cortical networks that become engaged to create the subjective experience during active gameplay. HIGHLIGHTSFirst study to investigate neural correlates of multiple dimensions of subjective gaming experience.Gameplay was manipulated with custom‐designed computer game targeting specific aspects of the subjective experience.Multivariate analyses identified brain patterns for subjective experiences of immersion, flow, challenge and negative affect.Additional searchlight analysis revealed brain patterns that can decode different game conditions.Results implicate a wide‐spread network of visual, default‐mode, and emotional areas for subjective gaming experience.

[1]  René Weber,et al.  Neural contributions to flow experience during video game playing. , 2012, Social cognitive and affective neuroscience.

[2]  Kent L. Norman,et al.  GEQ (Game Engagement/Experience Questionnaire): A Review of Two Papers , 2013, Interact. Comput..

[3]  Tao Liu,et al.  Easing and rising of tension from presence of others in player-observer turn-taking in a driving video game: A near-infrared spectroscopy study , 2012, CogSci.

[4]  Klaus-Robert Müller,et al.  Stereoscopic depth increases intersubject correlations of brain networks , 2014, NeuroImage.

[5]  Hans-Jochen Heinze,et al.  Different cortical activations for subjects using allocentric or egocentric strategies in a virtual navigation task , 2004, Neuroreport.

[6]  Karen Collins,et al.  Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design , 2008 .

[7]  H. Bless,et al.  Bulletin Personality and Social Psychology Flow and Regulatory Compatibility: an Experimental Approach to the Flow Model of Intrinsic Motivation on Behalf Of: Society for Personality and Social Psychology , 2022 .

[8]  M. Sahani,et al.  Cortical control of arm movements: a dynamical systems perspective. , 2013, Annual review of neuroscience.

[9]  G. Mangun,et al.  The neural mechanisms of top-down attentional control , 2000, Nature Neuroscience.

[10]  Glyn W. Humphreys,et al.  The Neural Underpinings of Simultanagnosia: Disconnecting the Visuospatial Attention Network , 2012, Journal of Cognitive Neuroscience.

[11]  A. Wunderlich,et al.  Brain activation during human navigation: gender-different neural networks as substrate of performance , 2000, Nature Neuroscience.

[12]  R. Viejo-Sobera,et al.  Neural Basis of Video Gaming: A Systematic Review , 2017, Front. Hum. Neurosci..

[13]  O. Blanke,et al.  Distinct illusory own-body perceptions caused by damage to posterior insula and extrastriate cortex. , 2013, Brain : a journal of neurology.

[14]  R. Hari,et al.  The opponent matters: elevated FMRI reward responses to winning against a human versus a computer opponent during interactive video game playing. , 2013, Cerebral cortex.

[15]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[16]  Luca Chittaro,et al.  Brain activity and prosocial behavior in a simulated life-threatening situation , 2014, NeuroImage.

[17]  James Paul Gee,et al.  What video games have to teach us about learning and literacy , 2007, CIE.

[18]  S. Lyubomirsky,et al.  The Benefits of Frequent Positive Affect: Does Happiness Lead to Success? , 2005, Psychological bulletin.

[19]  Lennart E. Nacke,et al.  Flow and immersion in first-person shooters: measuring the player's gameplay experience , 2008, Future Play.

[20]  A. Craig,et al.  How do you feel — now? The anterior insula and human awareness , 2009, Nature Reviews Neuroscience.

[21]  Marc N. Coutanche,et al.  Distinguishing Multi-voxel Patterns and Mean Activation: Why, How, and What Does It Tell Us? a Question of Spatial Frequency , 2022 .

[22]  John L. Sherry,et al.  Sex Differences in Video Game Play: , 2004, Commun. Res..

[23]  Debra K. Meyer,et al.  Students' Reports of Motivation and Negative Affect: A Theoretical and Empirical Analysis. , 1998 .

[24]  Lennart E. Nacke,et al.  More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game , 2010, Interact. Comput..

[25]  C. Wallraven,et al.  Visual and Haptic Shape Processing in the Human Brain: Unisensory Processing, Multisensory Convergence, and Top-Down Influences. , 2015, Cerebral cortex.

[26]  Sang Hee Kim,et al.  Neural Correlates of Positive and Negative Emotion Regulation , 2007, Journal of Cognitive Neuroscience.

[27]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[28]  Hailan Hu.  Reward and Aversion. , 2016, Annual review of neuroscience.

[29]  George A. Alvarez,et al.  The compensatory dynamic of inter-hemispheric interactions in visuospatial attention revealed using rTMS and fMRI , 2014, Front. Hum. Neurosci..

[30]  Xiaoqin Wang,et al.  Cortical Coding of Auditory Features. , 2018, Annual review of neuroscience.

[31]  S. Hillyard,et al.  Involvement of striate and extrastriate visual cortical areas in spatial attention , 1999, Nature Neuroscience.

[32]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[33]  A. Isen,et al.  The Influence of Positive Affect on Intrinsic and Extrinsic Motivation: Facilitating Enjoyment of Play, Responsible Work Behavior, and Self-Control , 2005 .

[34]  Frank M. Andrews,et al.  Social Indicators of Well-Being: Americans' Perceptions of Life Quality , 1976 .

[35]  R. Poldrack,et al.  Measuring neural representations with fMRI: practices and pitfalls , 2013, Annals of the New York Academy of Sciences.

[36]  I. Sprinkhuizen-Kuyper,et al.  DIFFICULTY SCALING OF GAME AI , 2004 .

[37]  GeeJames Paul.  What video games have to teach us about learning and literacy , 2003 .

[38]  J. Bulthé,et al.  Format-dependent representations of symbolic and non-symbolic numbers in the human cortex as revealed by multi-voxel pattern analyses , 2014, NeuroImage.

[39]  Lutz Jäncke,et al.  Neural Correlate of Spatial Presence in an Arousing and Noninteractive Virtual Reality: An EEG and Psychophysiology Study , 2006, Cyberpsychology Behav. Soc. Netw..

[40]  D. Hoang.  FLOW: The Psychology of Optimal Experience , 2018 .

[41]  L. Tyler,et al.  Understanding What We See: How We Derive Meaning From Vision , 2015, Trends in Cognitive Sciences.

[42]  Maurizio Corbetta,et al.  The human brain is intrinsically organized into dynamic, anticorrelated functional networks. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[43]  Russell A. Poldrack,et al.  Large-scale automated synthesis of human functional neuroimaging data , 2011, Nature Methods.

[44]  K. Zilles,et al.  The anatomical and functional specialization of the fusiform gyrus , 2016, Neuropsychologia.

[45]  A. Schleicher,et al.  Cytoarchitectonical analysis and probabilistic mapping of two extrastriate areas of the human posterior fusiform gyrus , 2012, Brain Structure and Function.

[46]  D. Watson.  Intraindividual and interindividual analyses of positive and negative affect: their relation to health complaints, perceived stress, and daily activities. , 1988, Journal of personality and social psychology.

[47]  G. Mangun,et al.  ERP and fMRI measures of visual spatial selective attention , 1998, Human brain mapping.

[48]  Emiliano Macaluso,et al.  Amygdala Activation Is Associated with Sense of Presence during Viewing 3D-surround Cinematography , 2013, ICONIP.

[49]  A. Kramer,et al.  Videogame training strategy-induced change in brain function during a complex visuomotor task , 2012, Behavioural Brain Research.

[50]  qcMIZCV QoKNQ,et al.  Increased Display Size and Resolution Improve Task Performance in Information-Rich Virtual Environments , 2006 .

[51]  M. Häusser,et al.  Cellular mechanisms of spatial navigation in the medial entorhinal cortex , 2013, Nature Neuroscience.

[52]  Donghun Chung,et al.  Influence of Gaming Display and Controller on Perceived Characteristics, Perceived Interactivity, Presence, and Discomfort , 2013, HCI.

[53]  H. Aarts,et al.  Positive affect as implicit motivator: on the nonconscious operation of behavioral goals. , 2005, Journal of personality and social psychology.

[54]  Christoph Klimmt,et al.  Explaining the enjoyment of playing video games: the role of competition , 2003, ICEC.

[55]  Peter Beike,et al.  Beyond Anova Basics Of Applied Statistics , 2016 .

[56]  Mariano Alcañiz Raya,et al.  Assessment of the influence of navigation control and screen size on the sense of presence in virtual reality using EEG , 2014, Expert Syst. Appl..

[57]  K. Amunts,et al.  Functional characterization and differential coactivation patterns of two cytoarchitectonic visual areas on the human posterior fusiform gyrus , 2014, Human brain mapping.

[58]  C. Pennartz.  Consciousness, Representation, Action: The Importance of Being Goal-Directed , 2018, Trends in Cognitive Sciences.

[59]  James V. Haxby,et al.  Multivariate pattern analysis of fMRI: The early beginnings , 2012, NeuroImage.

[60]  Brian Knutson,et al.  Anticipation of Increasing Monetary Reward Selectively Recruits Nucleus Accumbens , 2001, The Journal of Neuroscience.

[61]  Seong-Whan Lee,et al.  Abstract Representations of Associated Emotions in the Human Brain , 2015, The Journal of Neuroscience.

[62]  John Ashburner,et al.  A fast diffeomorphic image registration algorithm , 2007, NeuroImage.

[63]  Cristina Botella,et al.  An fMRI Study to Analyze Neural Correlates of Presence during Virtual Reality Experiences , 2014, Interact. Comput..

[64]  D. Kort,et al.  The Game Experience Questionnaire , 2013 .

[65]  Lutz Jäncke,et al.  Feeling Present in Arousing Virtual Reality Worlds: Prefrontal Brain Regions Differentially Orchestrate Presence Experience in Adults and Children , 2008, Frontiers in human neuroscience.

[66]  Martin P. Paulus,et al.  Accumulation of neural activity in the posterior insula encodes the passage of time , 2008, Neuropsychologia.

[67]  Johan Wagemans,et al.  Perceived Shape Similarity among Unfamiliar Objects and the Organization of the Human Object Vision Pathway , 2008, The Journal of Neuroscience.

[68]  K. Kallinen,et al.  Presence and emotion in computer game players during 1st person vs. 3rd person playing view: evidence from self-report, eye-tracking, and facial muscle activity data , 2007 .

[69]  Luca Chittaro,et al.  Neuroanatomical basis of concern-based altruism in virtual environment , 2017, Neuropsychologia.

[70]  Francisco Pereira,et al.  Information mapping with pattern classifiers: A comparative study , 2011, NeuroImage.

[71]  Christiane Waller,et al.  Neural correlates of experimentally induced flow experiences , 2014, NeuroImage.

[72]  René Weber,et al.  Think Aloud during fMRI: Neuronal Correlates of Subjective Experience in Video Games , 2008, Fun and Games.

[73]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[74]  Walter R. Boot,et al.  Effects of training strategies implemented in a complex videogame on functional connectivity of attentional networks , 2012, NeuroImage.

[75]  K. Ikoma,et al.  Brain activity during the flow experience: A functional near-infrared spectroscopy study , 2014, Neuroscience Letters.

[76]  Paul A. Cairns,et al.  Measuring and defining the experience of immersion in games , 2008, Int. J. Hum. Comput. Stud..

[77]  R. W. White.  Motivation reconsidered: the concept of competence. , 1959, Psychological review.

[78]  Itzhak Fried,et al.  Repeating Spatial Activations in Human Entorhinal Cortex , 2015, Current Biology.

[79]  P. Haggard,et al.  Neural signatures of body ownership: a sensory network for bodily self-consciousness. , 2007, Cerebral cortex.

[80]  D. Bavelier,et al.  Neural bases of selective attention in action video game players , 2012, Vision Research.

[81]  R. Hari,et al.  Just watching the game ain't enough: striatal fMRI reward responses to successes and failures in a video game during active and vicarious playing , 2013, Front. Hum. Neurosci..

[82]  P. Haggard,et al.  Having a body versus moving your body: Neural signatures of agency and body-ownership , 2010, Neuropsychologia.

[83]  Brian Knutson,et al.  Interactivity and Reward-Related Neural Activation during a Serious Videogame , 2012, PloS one.

[84]  René Weber,et al.  Neural networks underlying affective states in a multimodal virtual environment: contributions to boredom , 2013, Front. Hum. Neurosci..

[85]  A. Mayer,et al.  A functional MRI study of multimodal selective attention following mild traumatic brain injury , 2012, Brain Imaging and Behavior.