Improving HCI with Brain Input: Review, Trends, and Outlook

In the field of HCI, researchers from diverse backgrounds have taken a broad view of application domains that could benefit from brain signals, both by applying HCI methods to improve interfaces using brain signals (e.g., human-centered design and evaluation of brain-based user interfaces), as well as integrating brain signals into HCI methods (e.g., using brain metrics in user experience evaluation). Recent advances in brain sensing technologies, new analysis methods, and hardware improvements have opened the door for such research, which will accelerate with the increased commercialization of wearable technology containing brain sensors. In this monograph, we examine brain signals from an HCI perspective, focusing on work that makes an HCI-related contribution. We pursue three main goals. First, we give a primer for HCI researchers on the necessary technology, the possibilities, and limitations for using brain signals in user interfaces. Second, we systematically map out the research field by constructing a taxonomy of applications, input paradigms, and interface designs. For this purpose, we reviewed more than 100 publications in major HCI conferences and journals. Finally, we identify gaps and areas of emerging work to lay a foundation for future research on HCI for and with brain signals. Erin T. Solovey and Felix Putze (2021), “Improving HCI with Brain Input: Review, Trends, and Outlook”, Foundations and Trends® in Human–Computer Interaction: Vol. 13, No. 4, pp 298–379. DOI: 10.1561/1100000078. The version of record is available at: http://dx.doi.org/10.1561/1100000078

[1]  Tanja Schultz,et al.  Augmented Reality Interface for Smart Home Control using SSVEP-BCI and Eye Gaze , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[2]  Paul Cairns,et al.  Doing Better Statistics in Human-Computer Interaction , 2019 .

[3]  Edward Cutrell,et al.  BCI for passive input in HCI , 2007 .

[4]  Marc Cavazza,et al.  Anger-based BCI Using fNIRS Neurofeedback , 2015, UIST.

[5]  Marcus A. Magnor,et al.  EEG analysis of implicit human visual perception , 2012, CHI.

[6]  Matthew Pike,et al.  Examining the Reliability of Using fNIRS in Realistic HCI Settings for Spatial and Verbal Tasks , 2015, CHI.

[7]  Xuetong Zhai,et al.  The NIRS Brain AnalyzIR Toolbox , 2018, Algorithms.

[8]  Tanja Schultz,et al.  Investigating Intrusiveness of Workload Adaptation , 2014, ICMI.

[9]  Franck Tarpin-Bernard,et al.  Conceptual Priming for In-game BCI Training , 2015, TCHI.

[10]  Johan Eriksson,et al.  Effects of interactivity and 3D-motion on mental rotation brain activity in an immersive virtual environment , 2010, CHI.

[11]  Robert Oostenveld,et al.  The five percent electrode system for high-resolution EEG and ERP measurements , 2001, Clinical Neurophysiology.

[12]  Tanja Schultz,et al.  Multimodal person independent recognition of workload related biosignal patterns , 2011, ICMI '11.

[13]  Hongsong Li,et al.  Enhancing Audience Engagement in Performing Arts Through an Adaptive Virtual Environment with a Brain-Computer Interface , 2016, IUI.

[14]  Fotis Liarokapis,et al.  Examining and Enhancing the Illusory Touch Perception in Virtual Reality Using Non-Invasive Brain Stimulation , 2019, CHI.

[15]  Roberto Togneri,et al.  Quantifying target spotting performances with complex geoscientific imagery using ERP P300 responses , 2014, Int. J. Hum. Comput. Stud..

[16]  Jung-Tai King,et al.  An EEG-based Approach for Evaluating Graphic Icons from the Perspective of Semantic Distance , 2016, CHI.

[17]  Tanja Schultz,et al.  EEG-based Speech Recognition - Impact of Temporal Effects , 2009, BIOSIGNALS.

[18]  Klaus-Robert Müller,et al.  Introduction to machine learning for brain imaging , 2011, NeuroImage.

[19]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[20]  Mohan S. Kankanhalli,et al.  EEG-based Evaluation of Cognitive Workload Induced by Acoustic Parameters for Data Sonification , 2018, ICMI.

[21]  Riccardo Poli,et al.  Towards cooperative brain-computer interfaces for space navigation , 2013, IUI '13.

[22]  Albrecht Schmidt,et al.  Implicit human computer interaction through context , 2000, Personal Technologies.

[23]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[24]  Jung-Tai King,et al.  To Repeat or Not to Repeat?: Redesigning Repeating Auditory Alarms Based on EEG Analysis , 2019, CHI.

[25]  John Paulin Hansen,et al.  New technological windows into mind: there is more in eyes and brains for human-computer interaction , 1996, CHI.

[26]  Sarah Sharples,et al.  Measuring Mental Workload Variations in Office Work Tasks using fNIRS , 2021, Int. J. Hum. Comput. Stud..

[27]  Gui-Bin Bian,et al.  Removal of Artifacts from EEG Signals: A Review , 2019, Sensors.

[28]  Max L. Wilson,et al.  Using fNIRS in Usability Testing: Understanding the Effect of Web Form Layout on Mental Workload , 2016, CHI.

[29]  D. Boas,et al.  HomER: a review of time-series analysis methods for near-infrared spectroscopy of the brain. , 2009, Applied optics.

[30]  Nick Bryan-Kinns,et al.  Using affective and behavioural sensors to explore aspects of collaborative music making , 2015, Int. J. Hum. Comput. Stud..

[31]  Desney S. Tan,et al.  Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction , 2010 .

[32]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[33]  Jung-Tai King,et al.  Measuring the Influences of Musical Parameters on Cognitive and Behavioral Responses to Audio Notifications Using EEG and Large-scale Online Studies , 2019, CHI.

[34]  Xingda Qu,et al.  Affect prediction from physiological measures via visual stimuli , 2011, Int. J. Hum. Comput. Stud..

[35]  Daniel Afergan,et al.  Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State , 2016, CHI.

[36]  Martin Luessi,et al.  MNE software for processing MEG and EEG data , 2014, NeuroImage.

[37]  Harold W. Thimbleby,et al.  User experience evaluation through the brain's electrical activity , 2014, NordiCHI.

[38]  Thorsten O. Zander,et al.  Towards BCI-Based Implicit Control in Human–Computer Interaction , 2014 .

[39]  Alexander Kaplan,et al.  Adapting the P300-Based Brain–Computer Interface for Gaming: A Review , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[40]  Harry L. Graber,et al.  nirsLAB: A Computing Environment for fNIRS Neuroimaging Data Analysis , 2014 .

[41]  Brendan Z. Allison,et al.  Is It Significant? Guidelines for Reporting BCI Performance , 2012 .

[42]  Heinrich H. Bülthoff,et al.  Use the Right Sound for the Right Job: Verbal Commands and Auditory Icons for a Task-Management System Favor Different Information Processes in the Brain , 2018, CHI.

[43]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[44]  Fabien Lotte,et al.  A Tutorial on EEG Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces , 2014 .

[45]  José Miguel Morales,et al.  Validation of Electroencephalographic Recordings Obtained with a Consumer-Grade, Single Dry Electrode, Low-Cost Device: A Comparative Study , 2019, Sensors.

[46]  Daniel Afergan,et al.  Designing Implicit Interfaces for Physiological Computing , 2015, ACM Trans. Comput. Hum. Interact..

[47]  Chi Thanh Vi,et al.  Detecting error-related negativity for interaction design , 2012, CHI.

[48]  Anton Nijholt,et al.  Brain-Computer Interfaces Handbook: Technological and Theoretical Advances , 2018 .

[49]  Jérémy Frey,et al.  Framework for Electroencephalography-based Evaluation of User Experience , 2016, CHI.

[50]  Robert J. K. Jacob,et al.  This is your brain on interfaces: enhancing usability testing with functional near-infrared spectroscopy , 2011, CHI.

[51]  Desney S. Tan,et al.  Using a low-cost electroencephalograph for task classification in HCI research , 2006, UIST.

[52]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[53]  Giulio Jacucci,et al.  No Need to Laugh Out Loud , 2017, ACM Trans. Comput. Hum. Interact..

[54]  Tanja Schultz,et al.  Automatic classification of auto-correction errors in predictive text entry based on EEG and context information , 2017, ICMI.

[55]  Bilge Mutlu,et al.  Pay attention!: designing adaptive agents that monitor and improve user engagement , 2012, CHI.

[56]  Daniel Afergan,et al.  Dynamic difficulty using brain metrics of workload , 2014, CHI.

[57]  Rytis Maskeliunas,et al.  Distributed under Creative Commons Cc-by 4.0 Consumer-grade Eeg Devices: Are They Usable for Control Tasks? , 2022 .

[58]  Klaus-Robert Müller,et al.  Interpretable deep neural networks for single-trial EEG classification , 2016, Journal of Neuroscience Methods.

[59]  Gilbert Cockton,et al.  Mixing oil and water: transcending method boundaries in assistive technology for traumatic brain injury , 2000, CUU '00.

[60]  J. E. Korteling,et al.  Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls , 2015, Front. Neurosci..

[61]  Thorsten O. Zander,et al.  Enhancing Human-Computer Interaction with Input from Active and Passive Brain-Computer Interfaces , 2010, Brain-Computer Interfaces.

[62]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[63]  Raja Parasuraman,et al.  Neuroergonomics: The Brain at Work , 2006 .

[64]  Jeremy Grant,et al.  Commercial wireless versus standard stationary EEG systems for personalized emotional brain-computer interfaces: a preliminary reliability check , 2019, Neuroscience Research Notes.

[65]  Yijun Wang,et al.  Combining Brain-Computer Interface and Eye Tracking for High-Speed Text Entry in Virtual Reality , 2018, IUI.

[66]  Tonio Ball,et al.  A brain-computer interface for high-level remote control of an autonomous, reinforcement-learning-based robotic system for reaching and grasping , 2014, IUI.

[67]  Chi Thanh Vi,et al.  Neuroanatomical Correlates of Perceived Usability , 2017, UIST.

[68]  Raja Parasuraman,et al.  Wearable functional near infrared spectroscopy (fNIRS) and transcranial direct current stimulation (tDCS): expanding vistas for neurocognitive augmentation , 2015, Front. Syst. Neurosci..

[69]  Jie Liu,et al.  FOCUS: enhancing children's engagement in reading by using contextual BCI training sessions , 2014, CHI.

[70]  Robert J. K. Jacob,et al.  Eye Movement-Based Human-Computer Interaction Techniques: Toward Non-Command Interfaces , 2003 .

[71]  Satoshi Nakamura,et al.  Tracking liking state in brain activity while watching multiple movies , 2017, ICMI.

[72]  Anatole Lécuyer,et al.  A performance model of selection techniques for p300-based brain-computer interfaces , 2009, CHI.

[73]  Joseph G. Makin,et al.  Real-time decoding of question-and-answer speech dialogue using human cortical activity , 2019, Nature Communications.

[74]  José del R. Millán,et al.  Brain-Computer Interfaces , 2020, Handbook of Clinical Neurology.

[75]  Paulo Barraza,et al.  Implementing EEG hyperscanning setups , 2019, MethodsX.

[76]  J. Cacioppo,et al.  Inferring psychological significance from physiological signals. , 1990, The American psychologist.

[77]  Francisco J. Pelayo,et al.  Trends in EEG-BCI for daily-life: Requirements for artifact removal , 2017, Biomed. Signal Process. Control..

[78]  Jonathan R Wolpaw,et al.  Independent home use of a brain-computer interface by people with amyotrophic lateral sclerosis , 2018, Neurology.

[79]  E. Musk An Integrated Brain-Machine Interface Platform With Thousands of Channels , 2019, bioRxiv.

[80]  James Tompkin,et al.  A novel brain-computer interface using a multi-touch surface , 2010, CHI.

[81]  Tanja Schultz,et al.  Hybrid fNIRS-EEG based classification of auditory and visual perception processes , 2014, Front. Neurosci..

[82]  Roderick Murray-Smith,et al.  Simulating the feel of brain-computer interfaces for design, development and social interaction , 2011, CHI.

[83]  M. Nitsche,et al.  Excitability changes induced in the human motor cortex by weak transcranial direct current stimulation , 2000, The Journal of physiology.

[84]  Guillaume Gibert,et al.  OpenViBE: An Open-Source Software Platform to Design, Test, and Use BrainComputer Interfaces in Real and Virtual Environments , 2010, PRESENCE: Teleoperators and Virtual Environments.

[85]  E. Hildt Multi-Person Brain-To-Brain Interfaces: Ethical Issues , 2019, Front. Neurosci..

[86]  Matthias Scheutz,et al.  Brainput: enhancing interactive systems with streaming fnirs brain input , 2012, CHI.

[87]  Niels Henze,et al.  EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography , 2017, CHI.

[88]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[89]  Jarrod A. Lewis-Peacock,et al.  Closed-loop brain training: the science of neurofeedback , 2017, Nature Reviews Neuroscience.

[90]  Giulio Jacucci,et al.  The Psychophysiology Primer: A Guide to Methods and a Broad Review with a Focus on Human-Computer Interaction , 2016, Found. Trends Hum. Comput. Interact..

[91]  Bonnie Brinton Anderson,et al.  How Polymorphic Warnings Reduce Habituation in the Brain: Insights from an fMRI Study , 2015, CHI.

[92]  Shirley Coyle,et al.  On the suitability of near-infrared (NIR) systems for next-generation brain-computer interfaces. , 2004, Physiological measurement.

[93]  Chin-Teng Lin,et al.  Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials , 2019, CHI.

[94]  M.E. Davies,et al.  Source separation using single channel ICA , 2007, Signal Process..

[95]  Michael X Cohen,et al.  Analyzing Neural Time Series Data: Theory and Practice , 2014 .

[96]  Athanasios V. Vasilakos,et al.  Brain computer interface: control signals review , 2017, Neurocomputing.

[97]  Christian Kothe,et al.  Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general , 2011, Journal of neural engineering.

[98]  Robert J. K. Jacob,et al.  Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload , 2009, HCI.

[99]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[100]  Li-Wei Ko,et al.  An EEG-based approach for evaluating audio notifications under ambient sounds , 2014, CHI.

[101]  Anton Nijholt,et al.  BCIforReal: An application-Oriented Approach to BCI Out of the Laboratory , 2017, IUI Companion.

[102]  Ara W. Darzi,et al.  Imperial College near infrared spectroscopy neuroimaging analysis framework , 2017, Neurophotonics.

[103]  Christa Neuper,et al.  Using auditory event-related EEG potentials to assess presence in virtual reality , 2012, Int. J. Hum. Comput. Stud..

[104]  Roderick Murray-Smith,et al.  Hex: Dynamics and Probabilistic Text Entry , 2003, European Summer School on Multi-AgentControl.

[105]  Anton Nijholt,et al.  Brain–Computer Interface Software: A Review and Discussion , 2020, IEEE Transactions on Human-Machine Systems.

[106]  Chris Berka,et al.  Comparison of Medical and Consumer Wireless EEG Systems for Use in Clinical Trials , 2017, Front. Hum. Neurosci..

[107]  Senem Velipasalar,et al.  Building predictive models of emotion with functional near-infrared spectroscopy , 2018, Int. J. Hum. Comput. Stud..

[108]  Tanja Schultz,et al.  Intervention-free selection using EEG and eye tracking , 2016, ICMI.

[109]  Fabio Babiloni,et al.  The Dry Revolution: Evaluation of Three Different EEG Dry Electrode Types in Terms of Signal Spectral Features, Mental States Classification and Usability , 2019, Sensors.

[110]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing to evaluate information visualization interfaces , 2013, CHI.

[111]  Desney S. Tan,et al.  Brain-Computer Interfaces and Human-Computer Interaction , 2010, Brain-Computer Interfaces.

[112]  Hasan Ayaz,et al.  NEUROERGONOMICS , 2021, Handbook of Human Factors and Ergonomics.

[113]  Jérémy Frey,et al.  Teegi: tangible EEG interface , 2014, UIST.

[114]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[115]  José del R. Millán,et al.  Evaluation Criteria for BCI Research , 2007 .

[116]  Tanja Schultz,et al.  Brain-to-text: decoding spoken phrases from phone representations in the brain , 2015, Front. Neurosci..

[117]  Jinrui Zhang,et al.  FC-NIRS: A Functional Connectivity Analysis Tool for Near-Infrared Spectroscopy Data , 2015, BioMed research international.

[118]  Klaus-Robert Müller,et al.  Open Access Dataset for EEG+NIRS Single-Trial Classification , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[119]  Owen Falzon,et al.  A comparison of a broad range of EEG acquisition devices – is there any difference for SSVEP BCIs? , 2018, Brain-Computer Interfaces.

[120]  Tanja Schultz,et al.  Starring into the void?: Classifying Internal vs. External Attention from EEG , 2016, NordiCHI.

[121]  A. Pope,et al.  Biocybernetic system evaluates indices of operator engagement in automated task , 1995, Biological Psychology.

[122]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines , 2009, UIST '09.

[123]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[124]  Stephen H. Fairclough,et al.  Use of auditory event-related potentials to measure immersion during a computer game , 2015, Int. J. Hum. Comput. Stud..

[125]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[126]  Luca Mainardi,et al.  Performance measurement for brain–computer or brain–machine interfaces: a tutorial , 2014, Journal of neural engineering.

[127]  Aljo Mujcic,et al.  Mental workload vs. stress differentiation using single-channel EEG , 2017 .

[128]  A. Barker,et al.  NON-INVASIVE MAGNETIC STIMULATION OF HUMAN MOTOR CORTEX , 1985, The Lancet.

[129]  Eloisa Vargiu,et al.  Brain–Computer Interfaces on Track to Home: Results of the Evaluation at Disabled End-Users’ Homes and Lessons Learnt , 2015, Front. ICT.

[130]  Mary L. Cummings,et al.  Investigating Mental Workload Changes in a Long Duration Supervisory Control Task , 2015, Interact. Comput..

[131]  Robert J. K. Jacob,et al.  Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy , 2009, CHI.

[132]  Daniel Afergan,et al.  Brain-based target expansion , 2014, UIST.

[133]  Giulio Jacucci,et al.  Physiological Computing , 2015, Computer.

[134]  F Babiloni,et al.  Passive BCI beyond the lab: current trends and future directions , 2018, Physiological measurement.

[135]  Damien Coyle,et al.  A Review of Rapid Serial Visual Presentation-based Brain-1 Computer Interfaces 2 , 2017 .

[136]  Bertrand Rivet,et al.  Adding Human Learning in Brain--Computer Interfaces (BCIs) , 2015, ACM Trans. Comput. Hum. Interact..

[137]  Matteo Matteucci,et al.  A predictive speller controlled by a brain-computer interface based on motor imagery , 2012, TCHI.

[138]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[139]  Felix Putze,et al.  Methods and Tools for Using BCI with Augmented and Virtual Reality , 2019, Brain Art.

[140]  Kenneth Kreutz-Delgado,et al.  ICLabel: An automated electroencephalographic independent component classifier, dataset, and website , 2019, NeuroImage.

[141]  Guido Makransky,et al.  Measuring presence in video games: An investigation of the potential use of physiological measures as indicators of presence , 2019, Int. J. Hum. Comput. Stud..

[142]  Matthias Scheutz,et al.  Sensing cognitive multitasking for a brain-based adaptive user interface , 2011, CHI.

[143]  Alissa Nicole Antle,et al.  Opening up the Design Space of Neurofeedback Brain--Computer Interfaces for Children , 2018, ACM Trans. Comput. Hum. Interact..

[144]  Mihaly Benda,et al.  Brain–Computer Interface Spellers: A Review , 2018, Brain sciences.

[145]  Michael Goesele,et al.  How Human Am I?: EEG-based Evaluation of Virtual Characters , 2017, CHI.

[146]  Benjamin Blankertz,et al.  Designing for uncertain, asymmetric control: Interaction design for brain-computer interfaces , 2009, Int. J. Hum. Comput. Stud..

[147]  Rajesh P. N. Rao,et al.  BrainNet: A Multi-Person Brain-to-Brain Interface for Direct Collaboration Between Brains , 2018, ArXiv.

[148]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[149]  Sidney K. D'Mello,et al.  A Review and Meta-Analysis of Multimodal Affect Detection Systems , 2015, ACM Comput. Surv..

[150]  Gilbert Cockton,et al.  The "Cyberlink" Brain-Body Interface as an Assistive Technology for Persons with Traumatic Brain Injury: Longitudinal Results from a Group of Case Studies , 1999, Cyberpsychology Behav. Soc. Netw..

[151]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[152]  Reinhold Scherer,et al.  Mind the Traps! Design Guidelines for Rigorous BCI Experiments , 2018 .

[153]  Rajesh P. N. Rao,et al.  A Direct Brain-to-Brain Interface in Humans , 2014, PloS one.

[154]  Gilbert Cockton,et al.  Improving the performance of the cyberlink mental interface with “yes / no program” , 2001, CHI.

[155]  Karim Jerbi,et al.  Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy , 2015, Journal of Neuroscience Methods.

[156]  Peta Wyeth,et al.  Cooperative Game Play with Avatars and Agents: Differences in Brain Activity and the Experience of Play , 2015, CHI.