Toward an Open Data Repository and Meta-Analysis of Cognitive Data Using fNIRS Studies of Emotion

HCI research has increasingly incorporated the use of neurophysiological sensors to identify users’ cognitive and affective states. However, a persistent problem in machine learning on cognitive data is generalizability across participants. A proposed solution has been aggregating cognitive and survey data across studies to generate higher sample populations for machine learning and statistical analyses to converge in stable, generalizable results. In this paper, I argue that large data-sharing projects can facilitate the aggregation of results of brain imaging studies to address these issues, by smoothing noise in high-dimensional datasets. This paper contributes a small step towards large cognitive data sharing systems-design by proposing methods that facilitate the merging of currently incompatible fNIRS and FMRI datasets through term-based metadata analysis. To that end, I analyze 20 fNIRS studies of emotion using content analysis for: (1) synonym terms and definitions for ‘emotion,’ (2) the experimental stimuli, and (3) the use or non-use of self-report surveys. Results suggest that fNIRS studies of emotion have stable synonymy, using technical and folk conceptualizations of affective terms within and between publications to refer to emotion. The studies use different stimuli to elicit emotion but also show commonalities between shared use of standardized stimuli materials and self-report surveys. These similarities in conceptual synonymy and standardized experiment materials indicate promise for neuroimaging communities to establish open-data repositories based on metadata term-based analyses. This work contributes to efforts toward merging datasets across studies and between labs, unifying new modalities in neuroimaging such as fNIRS with fMRI datasets, increasing generalizability of machine learning models, and promoting the acceleration of science through open data-sharing infrastructure.

[1]  Martin A. Giese,et al.  Show me how you walk and I tell you how you feel — A functional near-infrared spectroscopy study on emotion perception based on human gait , 2014, NeuroImage.

[2]  Beatriz Luna,et al.  fNIRS evidence of prefrontal regulation of frustration in early childhood , 2014, NeuroImage.

[3]  Allen L. Edwards,et al.  The Relationship Between the Judged Desirability of a Trait and the Probability That the Trait Will Be Endorsed , 1953 .

[4]  E Altenmüller,et al.  Language processing in aphasia: changes in lateralization patterns during recovery reflect cerebral plasticity in adults. , 1997, Electroencephalography and clinical neurophysiology.

[5]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[6]  Thomas Dresler,et al.  The Effect of Emotional Content on Brain Activation and the Late Positive Potential in a Word n-back Task , 2013, PloS one.

[7]  Karim S. Kassam,et al.  The Effects of Measuring Emotion: Physiological Reactions to Emotional Situations Depend on whether Someone Is Asking , 2013, PloS one.

[8]  Helvi Kyngäs,et al.  The qualitative content analysis process. , 2008, Journal of advanced nursing.

[9]  Kamryn T. Eddy,et al.  Amygdala-frontal connectivity during emotion regulation. , 2007, Social cognitive and affective neuroscience.

[10]  Kotaro Takeda,et al.  Cerebral laterality differences in handedness: A mental rotation study with NIRS , 2008, Neuroscience Letters.

[11]  Markus J. Hofmann,et al.  The Role of Orbitofrontal Cortex in Processing Empathy Stories in 4- to 8-Year-Old Children , 2010, Front. Psychology.

[12]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[13]  P. Ekman Facial expression and emotion. , 1993, The American psychologist.

[14]  M. Balconi,et al.  Resting lateralized activity predicts the cortical response and appraisal of emotions: an fNIRS study. , 2015, Social cognitive and affective neuroscience.

[15]  Paul Pauli,et al.  Prefrontal Brain Activation During Emotional Processing: A Functional Near Infrared Spectroscopy Study (fNIRS) , 2011, The open neuroimaging journal.

[16]  Wolzt,et al.  World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. , 2003, The Journal of the American College of Dentists.

[17]  M. Just,et al.  Identifying Emotions on the Basis of Neural Activation , 2013, PloS one.

[18]  A. Jacobs,et al.  The Berlin Affective Word List Reloaded (BAWL-R) , 2009, Behavior research methods.

[19]  S. Fairclough,et al.  Activation of the rostromedial prefrontal cortex during the experience of positive emotion in the context of esthetic experience. An fNIRS study , 2013, Frontiers in Human Neuroscience.

[20]  S. Bunce,et al.  Functional near-infrared spectroscopy , 2006, IEEE Engineering in Medicine and Biology Magazine.

[21]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[22]  Janet B W Williams,et al.  A structured interview guide for the Hamilton Depression Rating Scale. , 1988, Archives of general psychiatry.

[23]  Stephan Hamann,et al.  Sex differences in brain activation to emotional stimuli: A meta-analysis of neuroimaging studies , 2012, Neuropsychologia.

[24]  M. Rothbart,et al.  Investigations of temperament at three to seven years: the Children's Behavior Questionnaire. , 2001, Child development.

[25]  Maurizio Codispoti,et al.  Watching emotional movies: affective reactions and gender differences. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[26]  Yoko Mano,et al.  Decoding what one likes or dislikes from single-trial fNIRS measurements , 2011, Neuroreport.

[27]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[28]  S Nioka,et al.  Quantitation of time- and frequency-resolved optical spectra for the determination of tissue oxygenation. , 1991, Analytical biochemistry.

[29]  R. Lane,et al.  Claude Bernard and the heart–brain connection: Further elaboration of a model of neurovisceral integration , 2009, Neuroscience & Biobehavioral Reviews.

[30]  Martin Wolf,et al.  Prefrontal cortex activity, sympatho-vagal reaction and behaviour distinguish between situations of feed reward and frustration in dwarf goats , 2013, Behavioural Brain Research.

[31]  Ann-Christine Ehlis,et al.  Inhibitory transcranial magnetic theta burst stimulation attenuates prefrontal cortex oxygenation , 2013, Human brain mapping.

[32]  Alan C. Evans,et al.  Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions , 1999, Nature Neuroscience.

[33]  Vir V. Phoha,et al.  fNIRS: A new modality for brain activity-based biometric authentication , 2015, 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[34]  Giulio Jacucci,et al.  Everyday Life as a Stage in Creating and Performing Scenarios for Wireless Devices , 2002, Personal and Ubiquitous Computing.

[35]  Giuseppe Riva,et al.  Emotional Response to Virtual Reality Exposure across Different Cultures: The Role of the Attribution Process , 2009, Cyberpsychology Behav. Soc. Netw..

[36]  J. Leon-Carrion,et al.  A lasting post-stimulus activation on dorsolateral prefrontal cortex is produced when processing valence and arousal in visual affective stimuli , 2007, Neuroscience Letters.

[37]  K. Luan Phan,et al.  Functional Neuroanatomy of Emotion: A Meta-Analysis of Emotion Activation Studies in PET and fMRI , 2002, NeuroImage.

[38]  Matt Germonprez,et al.  The Open Community Data Exchange: Advancing Data Sharing and Discovery in Open Online Community Science , 2016, OpenSym.

[39]  R. C. Young,et al.  A Rating Scale for Mania: Reliability, Validity and Sensitivity , 1978, British Journal of Psychiatry.

[40]  Rosalind W. Picard,et al.  Affective Computing and Autism , 2006, Annals of the New York Academy of Sciences.

[41]  Yumie Ono,et al.  fMRI Validation of fNIRS Measurements During a Naturalistic Task , 2015, Journal of visualized experiments : JoVE.

[42]  Kurtulus Izzetoglu,et al.  Differential time course and intensity of PFC activation for men and women in response to emotional stimuli: A functional near-infrared spectroscopy (fNIRS) study , 2006, Neuroscience Letters.

[43]  A. Ehlis,et al.  Prefrontal activation patterns of automatic and regulated approach–avoidance reactions – A functional near-infrared spectroscopy (fNIRS) study , 2013, Cortex.

[44]  L. R. Altimari,et al.  How Motivational and Calm Music May Affect the Prefrontal Cortex Area and Emotional Responses: A Functional Near-Infrared Spectroscopy (fNIRS) Study , 2015, Perceptual and motor skills.

[45]  R. Adolphs,et al.  Cortical Systems for the Recognition of Emotion in Facial Expressions , 1996, The Journal of Neuroscience.

[46]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[47]  Claus Tempelmann,et al.  Are emotions associated with activity during rest or interoception? An exploratory fMRI study in healthy subjects , 2011, Neuroscience Letters.

[48]  J. Thayer,et al.  eview meta-analysis of heart rate variability and neuroimaging studies : Implications or heart rate variability as a marker of stress and health , 2012 .

[49]  S. Rauch,et al.  Response and Habituation of the Human Amygdala during Visual Processing of Facial Expression , 1996, Neuron.

[50]  Toshihisa Tanaka,et al.  Interactive component extraction from fEEG, fNIRS and peripheral biosignals for affective brain-machine interfacing paradigms , 2011, Comput. Hum. Behav..

[51]  Mark R. Costa,et al.  Truthiness: Challenges Associated with Employing Machine Learning on Neurophysiological Sensor Data , 2016, HCI.

[52]  D. Perrett,et al.  Dissociable neural responses to facial expressions of sadness and anger. , 1999, Brain : a journal of neurology.

[53]  Michael B. Hudson,et al.  Examining How Gender and Emoticons Influence Facebook Jealousy , 2015, Cyberpsychology Behav. Soc. Netw..

[54]  Zhen Yuan,et al.  Fusion of fNIRS and fMRI data: identifying when and where hemodynamic signals are changing in human brains , 2013, Front. Hum. Neurosci..

[55]  K. Scherer What are emotions? And how can they be measured? , 2005 .

[56]  K. Ikoma,et al.  Brain activity during the flow experience: A functional near-infrared spectroscopy study , 2014, Neuroscience Letters.

[57]  E. Goffman The Presentation of Self in Everyday Life , 1959 .

[58]  Mohammad Soleymani,et al.  Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection , 2016, IEEE Transactions on Affective Computing.

[59]  M. Yücel,et al.  Sex differences in the neural correlates of emotion: Evidence from neuroimaging , 2011, Biological Psychology.

[60]  R. Lanius,et al.  The neurobiology of emotion regulation in posttraumatic stress disorder: Amygdala downregulation via real‐time fMRI neurofeedback , 2017, Human brain mapping.

[61]  Henrik M. Peperkorn,et al.  The impact of perception and presence on emotional reactions: a review of research in virtual reality , 2015, Front. Psychol..