Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls

Estimating cognitive or affective state from neurophysiological signals and designing applications that make use of this information requires expertise in many disciplines such as neurophysiology, machine learning, experimental psychology, and human factors. This makes it difficult to perform research that is strong in all its aspects as well as to judge a study or application on its merits. On the occasion of the special topic “Using neurophysiological signals that reflect cognitive or affective state” we here summarize often occurring pitfalls and recommendations on how to avoid them, both for authors (researchers) and readers. They relate to defining the state of interest, the neurophysiological processes that are expected to be involved in the state of interest, confounding factors, inadvertently “cheating” with classification analyses, insight on what underlies successful state estimation, and finally, the added value of neurophysiological measures in the context of an application. We hope that this paper will support the community in producing high quality studies and well-validated, useful applications.

[1]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[2]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[3]  J. Cacioppo,et al.  Inferring psychological significance from physiological signals. , 1990, The American psychologist.

[4]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[5]  Glenn F. Wilson,et al.  Psychophysiological responses to changes in workload during simulated air traffic control , 1996, Biological Psychology.

[6]  W. Klimesch EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis , 1999, Brain Research Reviews.

[7]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[8]  M. Bradley,et al.  Affective reactions to briefly presented pictures. , 2001, Psychophysiology.

[9]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[10]  Martha J. Farah,et al.  Emerging ethical issues in neuroscience , 2002, Nature Neuroscience.

[11]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[12]  T. Canli,et al.  Neuroimaging of emotion and personality: Scientific evidence and ethical considerations , 2002, Brain and Cognition.

[13]  S. Dickerson,et al.  Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. , 2004, Psychological bulletin.

[14]  Jdel.R. Millan,et al.  On the need for on-line learning in brain-computer interfaces , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[15]  C. Neuper,et al.  EEG alpha band dissociation with increasing task demands. , 2005, Brain research. Cognitive brain research.

[16]  Raja Parasuraman,et al.  Neuroergonomics: The Brain at Work , 2006 .

[17]  G. Pfurtscheller,et al.  Graz brain-computer interface II: towards communication between humans and computers based on online classification of three different EEG patterns , 1996, Medical and Biological Engineering and Computing.

[18]  Yonghong Huang,et al.  A Fusion Approach for Image Triage using Single Trial ERP Detection , 2007, 2007 3rd International IEEE/EMBS Conference on Neural Engineering.

[19]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[20]  E. Whitham,et al.  Scalp electrical recording during paralysis: Quantitative evidence that EEG frequencies above 20Hz are contaminated by EMG , 2007, Clinical Neurophysiology.

[21]  Clemens Brunner,et al.  Better than random? A closer look on BCI results , 2008 .

[22]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[23]  R. Schleicher,et al.  Blinks and saccades as indicators of fatigue in sleepiness warners: looking tired? , 2022 .

[24]  Deena Skolnick Weisberg,et al.  The Seductive Allure of Neuroscience Explanations , 2008, Journal of Cognitive Neuroscience.

[25]  P. Howard-Jones Scepticism is not enough , 2009, Cortex.

[26]  Michael D. Robinson,et al.  Measures of emotion: A review , 2009, Cognition & emotion.

[27]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[28]  S. Gielen,et al.  The brain–computer interface cycle , 2009, Journal of neural engineering.

[29]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[30]  T. Heskes,et al.  Covert attention allows for continuous control of brain–computer interfaces , 2010, The European journal of neuroscience.

[31]  David Girbau,et al.  ANALYSIS OF VITAL SIGNS MONITORING USING AN IR-UWB RADAR , 2010 .

[32]  Klas Ihme,et al.  A Dry EEG-System for Scientific Research and Brain–Computer Interfaces , 2010, Front. Neurosci..

[33]  Eva Oliveira,et al.  Towards Emotional Interaction: Using Movies to Automatically Learn Users' Emotional States , 2011, INTERACT.

[34]  Boris Reuderink,et al.  Robust Brain-Computer Interfaces , 2011 .

[35]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[36]  Christian Kothe,et al.  Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general , 2011, Journal of neural engineering.

[37]  Klaus-Robert Müller,et al.  Introduction to machine learning for brain imaging , 2011, NeuroImage.

[38]  T. Jung,et al.  Dry and Noncontact EEG Sensors for Mobile Brain–Computer Interfaces , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Fabien Lotte,et al.  Brain-Computer Interfaces: Beyond Medical Applications , 2012, Computer.

[40]  Pedro M. Domingos A few useful things to know about machine learning , 2012, Commun. ACM.

[41]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[42]  Mariano Sigman,et al.  Fixation-related potentials in visual search: a combined EEG and eye tracking study. , 2012, Journal of vision.

[43]  Robert Oostenveld,et al.  Estimating workload using EEG spectral power and ERPs in the n-back task , 2012, Journal of neural engineering.

[44]  Benjamin Schrauwen,et al.  A Bayesian Model for Exploiting Application Constraints to Enable Unsupervised Training of a P300-based BCI , 2012, PloS one.

[45]  David Vernon,et al.  A novel self-guided approach to alpha activity training. , 2012, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[46]  W. Klimesch Alpha-band oscillations, attention, and controlled access to stored information , 2012, Trends in Cognitive Sciences.

[47]  Wolfgang Rosenstiel,et al.  Using Cross-Task Classification for Classifying Workload Levels in Complex Learning Tasks , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[48]  Boris Reuderink,et al.  Distinguishing between target and nontarget fixations in a visual search task using fixation-related potentials. , 2013, Journal of vision.

[49]  Maarten A. Hogervorst,et al.  Physiological Correlates of Stress in Individuals about to Undergo Eye Laser Surgery , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[50]  Jan B. F. van Erp,et al.  Perceiving blocks of emotional pictures and sounds: effects on physiological variables , 2013, Front. Hum. Neurosci..

[51]  C. Keysers,et al.  EEG sensorimotor correlates of translating sounds into actions , 2013, Front. Neurosci..

[52]  Dick de Waard,et al.  Classifying visuomotor workload in a driving simulator using subject specific spatial brain patterns , 2013, Front. Neurosci..

[53]  Dirk Heylen,et al.  Effortless Passive BCIs for Healthy Users , 2013, HCI.

[54]  Stephen H. Fairclough,et al.  Capturing user engagement via psychophysiology: measures and mechanisms for biocybernetic adaptation , 2013, Int. J. Auton. Adapt. Commun. Syst..

[55]  A. Frolov,et al.  Motor Imagery and Its Practical Application , 2014, Neuroscience and Behavioral Physiology.

[56]  T. Jung,et al.  Developing an EEG-based on-line closed-loop lapse detection and mitigation system , 2014, Front. Neurosci..

[57]  John J. Furedy,et al.  Brain fingerprinting classification concealed information test detects US Navy military medical information with P300 , 2014, Front. Neurosci..

[58]  Ricardo Chavarriaga,et al.  Errare machinale est: the use of error-related potentials in brain-machine interfaces , 2014, Front. Neurosci..

[59]  K. Kashihara A brain-computer interface for potential non-verbal facial communication based on EEG signals related to specific emotions , 2014, Front. Neurosci..

[60]  Wolfgang Rosenstiel,et al.  Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach , 2014, Front. Neurosci..

[61]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[62]  T. Jung,et al.  Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening , 2014, Front. Neurosci..

[63]  Christian Mühl,et al.  EEG-based workload estimation across affective contexts , 2014, Front. Neurosci..

[64]  Stephen H. Fairclough,et al.  Advances in Physiological Computing , 2014, Human–Computer Interaction Series.

[65]  J. Buitelaar,et al.  Does EEG-neurofeedback improve neurocognitive functioning in children with attention-deficit/hyperactivity disorder? A systematic review and a double-blind placebo-controlled study. , 2014, Journal of child psychology and psychiatry, and allied disciplines.

[66]  René J. Huster,et al.  Self-regulation of frontal-midline theta facilitates memory updating and mental set shifting , 2014, Front. Behav. Neurosci..

[67]  Chin-Teng Lin,et al.  Knowledge-based identification of sleep stages based on two forehead electroencephalogram channels , 2014, Front. Neurosci..

[68]  Maarten De Vos,et al.  P300 speller BCI with a mobile EEG system: comparison to a traditional amplifier , 2014, Journal of neural engineering.

[69]  Jörn M. Horschig,et al.  Hypothesis-driven methods to augment human cognition by optimizing cortical oscillations , 2014, Front. Syst. Neurosci..

[70]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[71]  A. Stuiver,et al.  Cardiovascular state changes in simulated work environments , 2014, Front. Neurosci..

[72]  Matthias Scheutz,et al.  What we can and cannot (yet) do with functional near infrared spectroscopy , 2014, Front. Neurosci..

[73]  Maarten A. Hogervorst,et al.  A new paradigm to induce mental stress: the Sing-a-Song Stress Test (SSST) , 2014, Front. Neurosci..

[74]  Peter Werkhoven,et al.  GASICA: generic automated stress induction and control application design of an application for controlling the stress state , 2014, Front. Neurosci..

[75]  Chris Berka,et al.  Modeling temporal sequences of cognitive state changes based on a combination of EEG-engagement, EEG-workload, and heart rate metrics , 2014, Front. Neurosci..

[76]  Jan B. F. van Erp,et al.  Evidence for effects of task difficulty but not learning on neurophysiological variables associated with effort. , 2014, International Journal of Psychophysiology.

[77]  Jon Touryan,et al.  Estimating endogenous changes in task performance from EEG , 2014, Front. Neurosci..

[78]  J. Wolpaw,et al.  Brain-computer interface (BCI) evaluation in people with amyotrophic lateral sclerosis , 2014, Amyotrophic lateral sclerosis & frontotemporal degeneration.

[79]  Alessandro Tognetti,et al.  Inference of human affective states from psychophysiological measurements extracted under ecologically valid conditions , 2014, Front. Neurosci..

[80]  Tanja Schultz,et al.  Hybrid fNIRS-EEG based classification of auditory and visual perception processes , 2014, Front. Neurosci..

[81]  Alexander J. Casson,et al.  Artificial Neural Network classification of operator workload with an assessment of time variation and noise-enhancement to increase performance , 2014, Front. Neurosci..

[82]  James C. Christensen,et al.  Electrode replacement does not affect classification accuracy in dual-session use of a passive brain-computer interface for assessing cognitive workload , 2015, Front. Neurosci..

[83]  Dylan D. Schmorrow,et al.  Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience , 2016, Lecture Notes in Computer Science.