Brain-Computer Interfacing and Games

Recently research into Brain-Computer Interfacing (BCI) applications for healthy users, such as games, has been initiated. But why would a healthy person use a still-unproven technology such as BCI for game interaction? BCI provides a combination of information and features that no other input modality can offer. But for general acceptance of this technology, usability and user experience will need to be taken into account when designing such systems. Therefore, this chapter gives an overview of the state of the art of BCI in games and discusses the consequences of applying knowledge from Human-Computer Interaction (HCI) to the design of BCI for games. The integration of HCI with BCI is illustrated by research examples and showcases, intended to take this promising technology out of the lab. Future research needs to move beyond feasibility tests, to prove that BCI is also applicable in realistic, real-world settings.

[1]  Sara Ilstedt Hjelm,et al.  Brainball - using brain activity for cool competition , 2000 .

[2]  J.D. Bayliss,et al.  Use of the evoked potential P3 component for control in a virtual apartment , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[3]  Avi Rushinek,et al.  What makes users happy? , 1986, CACM.

[4]  Julie A. Jacko.  Novel interaction methods and techniques , 2009 .

[5]  Jeong-Hoon Shin,et al.  Implementation of a 3-Dimensional Game for developing balanced Brainwave , 2007, 5th ACIS International Conference on Software Engineering Research, Management & Applications (SERA 2007).

[6]  Christa Neuper,et al.  Walking from thoughts: Not the muscles are crucial, but the brain waves! , 2005 .

[7]  Melody Moore Jackson,et al.  Continuous Control Paradigms for Direct Brain Interfaces , 2009, HCI.

[8]  Jakob Nielsen,et al.  Usability Metrics: Tracking Interface Improvements , 1996, IEEE Softw..

[9]  Anton Nijholt,et al.  Emotional brain-computer interfaces , 2009, ACII.

[10]  Mercedes Atienza,et al.  Spectral Structure and Brain Mapping of Human Alpha Activities in Different Arousal States , 1999, Neuropsychobiology.

[11]  Lennart E. Nacke,et al.  Flow and immersion in first-person shooters: measuring the player's gameplay experience , 2008, Future Play.

[12]  Klaus-Robert Müller,et al.  Playing Pinball with non-invasive BCI , 2008, NIPS.

[13]  R. Burke,et al.  Brain Computer Interface based on the Steady-State VEP for Immersive Gaming Control , 2004 .

[14]  J. Pineda,et al.  Learning to control brain rhythms: making a brain-computer interface possible , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[15]  Matthias M. Müller,et al.  Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials , 2001, Clinical Neurophysiology.

[16]  Liqing Zhang,et al.  EEG-based asynchronous BCI control of a car in 3D virtual reality environments , 2009 .

[17]  Changchun Liu,et al.  Dynamic Difficulty Adjustment in Computer Games Through Real-Time Anxiety-Based Affective Feedback , 2009, Int. J. Hum. Comput. Interact..

[18]  J. Guardia,et al.  Feature Extraction for Multi-class BCI using Canonical Variates Analysis , 2007, 2007 IEEE International Symposium on Intelligent Signal Processing.

[19]  Matthias M. Müller,et al.  Processing of affective pictures modulates right-hemispheric gamma band EEG activity , 1999, Clinical Neurophysiology.

[20]  Paolo Bonato,et al.  Neuroadaptive technologies: Applying neuroergonomics to the design of advanced interfaces , 2003 .

[21]  Anton Nijholt,et al.  BrainGain: BCI for HCI and Games , 2008 .

[22]  Liu Mingyu,et al.  Development of EEG Biofeedback System Based on Virtual Reality Environment , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[23]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[24]  G. Pfurtscheller,et al.  How many people are able to operate an EEG-based brain-computer interface (BCI)? , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[25]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines , 2009, UIST '09.

[26]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Sara Ilstedt Hjelm,et al.  Research + design: the making of Brainball , 2003, INTR.

[28]  Jessica D. Bayliss,et al.  Changing the P300 Brain Computer Interface , 2004, Cyberpsychology Behav. Soc. Netw..

[29]  Reinhold Scherer,et al.  A fully on-line adaptive BCI , 2006, IEEE Transactions on Biomedical Engineering.

[30]  Michael W. Haas,et al.  Navigating through virtual flight environments using brain-body-actuated control , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[31]  Dirk Heylen,et al.  Actual and Imagined Movement in BCI Gaming , 2009 .

[32]  M. Hallett,et al.  Binary EEG Control for Two-Dimensional Cursor Movement: An Online Approach , 2007, 2007 IEEE/ICME International Conference on Complex Medical Engineering.

[33]  José del R. Millán,et al.  Error-Related EEG Potentials Generated During Simulated Brain–Computer Interaction , 2008, IEEE Transactions on Biomedical Engineering.

[34]  Yuan-Pin Lin,et al.  EEG-based emotion recognition in music listening: A comparison of schemes for multiclass support vector machine , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[35]  Mikko Sams,et al.  Online Classification of Single EEG Trials During Finger Movements , 2008, IEEE Transactions on Biomedical Engineering.

[36]  J.J. Vidal,et al.  Real-time detection of brain events in EEG , 1977, Proceedings of the IEEE.

[37]  D. Kort,et al.  D3.3 : Game Experience Questionnaire:development of a self-report measure to assess the psychological impact of digital games , 2007 .

[38]  T. Fernández,et al.  Broad Band Spectral Measurements of EEG During Emotional Tasks , 2001, The International journal of neuroscience.

[39]  Horst Bischof,et al.  The Self-Paced Graz Brain-Computer Interface: Methods and Applications , 2007, Comput. Intell. Neurosci..

[40]  K. Scherer,et al.  Psychophysiological responses to appraisal dimensions in a computer game , 2004 .

[41]  M. Csíkszentmihályi.  Flow. The Psychology of Optimal Experience. New York (HarperPerennial) 1990. , 1990 .

[42]  Dirk Heylen,et al.  Bacteria Hunt , 2010, Journal on Multimodal User Interfaces.

[43]  G Calhoun,et al.  Brain-computer interfaces based on the steady-state visual-evoked response. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[44]  Andrzej Cichocki,et al.  Fully Online Multicommand Brain-Computer Interface with Visual Neurofeedback Using SSVEP Paradigm , 2007, Comput. Intell. Neurosci..

[45]  L. Aftanas,et al.  Analysis of Evoked EEG Synchronization and Desynchronization in Conditions of Emotional Activation in Humans: Temporal and Topographic Characteristics , 2004, Neuroscience and Behavioral Physiology.

[46]  Marko Turpeinen,et al.  Emotionally Adapted Games - An Example of a First Person Shooter , 2009, HCI.

[47]  M. Hallett,et al.  What is the Bereitschaftspotential? , 2006, Clinical Neurophysiology.

[48]  Michitaka Hirose,et al.  Brain-Computer Interfaces, Virtual Reality, and Videogames , 2008, Computer.

[49]  N. Sobell,et al.  Brainwave Drawing Game , 1989, Delicate Balance: Technics, Culture and Consequences.

[50]  Manfred Tscheligi,et al.  Proceedings of the international conference on Advances in computer entertainment technology , 2007, ACE 2008.

[51]  John R. Smith,et al.  Steady-State VEP-Based Brain-Computer Interface Control in an Immersive 3D Gaming Environment , 2005, EURASIP J. Adv. Signal Process..

[52]  Jaimie F Borisoff,et al.  Real-Time Control of a Video Game With a Direct Brain–Computer Interface , 2004, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[53]  Thorsten O. Zander,et al.  Utilizing Secondary Input from Passive Brain-Computer Interfaces for Enhancing Human-Machine Interaction , 2009, HCI.

[54]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[55]  Michael Thornton Wyman,et al.  2 – World of Warcraft , 2011 .

[56]  Paras Kaul.  Neurological gaming environments , 2006, SIGGRAPH '06.

[57]  F. Lotte,et al.  Self-Paced Brain-Computer Interaction with Virtual Worlds: A Quantitative and Qualitative Study "Out of the Lab" , 2008 .

[58]  Motoaki Kawanabe,et al.  Invariant Common Spatial Patterns: Alleviating Nonstationarities in Brain-Computer Interfacing , 2007, NIPS.

[59]  Dirk Heylen,et al.  Cross-modal elicitation of affective experience , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[60]  Wolfgang Klimesch,et al.  EEG alpha power and intelligence , 2002 .

[61]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Dong Pyo Jang,et al.  Attention Enhancement System using virtual reality and EEG biofeedback , 2002, Proceedings IEEE Virtual Reality 2002.

[63]  Scott Stevens,et al.  Proceedings of the 7th International Conference on Entertainment Computing , 2008 .

[64]  R. Davidson.  Anterior cerebral asymmetry and the nature of emotion , 1992, Brain and Cognition.

[65]  Anton Nijholt,et al.  Affective Pacman: A Frustrating Game for Brain-Computer Interface Experiments , 2009, INTETAIN.

[66]  Desney S. Tan,et al.  Playing with your brain: brain-computer interfaces and games , 2007, ACE '07.

[67]  Chang Song,et al.  Development of a Neuron Based Internet Game Driven by a Brain-Computer Interface System , 2006, 2006 International Conference on Hybrid Information Technology.

[68]  John A. Mariani,et al.  Mind over virtual matter: using virtual environments for neurofeedback training , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[69]  Egon L. van den Broek,et al.  Prerequisites for Affective Signal Processing (ASP) , 2009, BIOSIGNALS.

[70]  P. Tyson,et al.  Task-related stress and EEG alpha biofeedback , 1987, Biofeedback and self-regulation.

[71]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[72]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[73]  Lucas Tschuor.  Computer Game Control through Relaxation-Induced EEG Changes , 2004 .

[74]  Desney S. Tan,et al.  Brain-Computer Interfacing for Intelligent Systems , 2008, IEEE Intelligent Systems.

[75]  Cuntai Guan,et al.  Introduction to NeuroComm: a Platform for Developing Real-Time EEG-based Brain-Computer Interface Applications , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[76]  Han-Jeong Hwang,et al.  Neurofeedback-based motor imagery training for brain–computer interface (BCI) , 2009, Journal of Neuroscience Methods.

[77]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[78]  Regan L. Mandryk,et al.  Using psychophysiological techniques to measure user experience with entertainment technologies , 2006, Behav. Inf. Technol..

[79]  Guillaume Chanel,et al.  Boredom, engagement and anxiety as indicators for adaptation to difficulty in games , 2008, MindTrek '08.

[80]  David Sander,et al.  A systems approach to appraisal mechanisms in emotion , 2005, Neural Networks.

[81]  Anton Nijholt,et al.  Turning Shortcomings into Challenges: Brain-Computer Interfaces for Games , 2009, INTETAIN.

[82]  G. Pfurtscheller,et al.  Walking through a virtual city by thought , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[83]  L. R. John,et al.  Quantifying Mental Relaxation with EEG for use in Computer Games , 2006, International Conference on Internet Computing.

[84]  Jeremy R. Cooperstock,et al.  Biosignals Analysis and its Application in a Performance Setting - Towards the Development of an Emotional-Imaging Generator , 2008, BIOSIGNALS.

[85]  Julie A. Jacko,et al.  Interacting in various application domains , 2009 .

[86]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[87]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[88]  Starr Roxanne Hiltz,et al.  User satisfaction with computer-mediated communication systems , 1990 .

[89]  Mohammad Obaid,et al.  "Feed the Fish": an affect-aware game , 2008, IE.

[90]  J. Wolpaw,et al.  Mu and Beta Rhythm Topographies During Motor Imagery and Actual Movements , 2004, Brain Topography.

[91]  A. Murat Tekalp,et al.  Multimedia Content Representation, Classification and Security, International Workshop, MRCS 2006, Istanbul, Turkey, September 11-13, 2006, Proceedings , 2006, MRCS.

[92]  Henry C. Lucas,et al.  Why information systems fail , 1975 .

[93]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[94]  Nancy C. Goodwin,et al.  Functionality and usability , 1987, CACM.

[95]  Guillaume Chanel,et al.  Emotion Assessment: Arousal Evaluation Using EEG's and Peripheral Physiological Signals , 2006, MRCS.

[96]  Wijnand A. IJsselsteijn,et al.  Creating an Emotionally Adaptive Game , 2008, ICEC.

[97]  Christian Müller-Tomfelde,et al.  Dwell-Based Pointing in Applications of Human Computer Interaction , 2007, INTERACT.

[98]  Helge J. Ritter,et al.  2009 Special Issue: The MindGame: A P300-based brain-computer interface game , 2009 .

[99]  Jon D. Morris.  Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[100]  Reinhold Scherer,et al.  Navigation in Virutal Environments through Motor Imagery , 2004 .

[101]  J D Bayliss,et al.  A virtual reality testbed for brain-computer interface research. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[102]  Luca Citi,et al.  Defining brain–machine interface applications by matching interface performance with device requirements , 2008, Journal of Neuroscience Methods.

[103]  Stephen H. Fairclough,et al.  Fundamentals of physiological computing , 2009, Interact. Comput..

[104]  Dirk Heylen,et al.  Bacteria Hunt: A multimodal, multiparadigm BCI game , 2010 .

[105]  Janice S. Aikins.  Out of the Lab , 1993, IEEE Expert.

[106]  Alan T. Pope,et al.  Helping Video Games Rewire "Our Minds" , 2001 .

[107]  Klaus-Robert Müller,et al.  The Berlin Brain-Computer Interface (BBCI) – towards a new communication channel for online control in gaming applications , 2007, Multimedia Tools and Applications.

[108]  Nadia Bianchi-Berthouze,et al.  Does Body Movement Engage You More in Digital Game Play? and Why? , 2007, ACII.

[109]  Blake Ives,et al.  The measurement of user information satisfaction , 1983, CACM.

[110]  Boris Reuderink,et al.  BrainBasher: a BCI Game , 2008 .