Brain-Computer Interface Systems Used for Virtual Reality Control

A Brain-Computer Interface (BCI) is a non-muscular communication channel for connecting the brain to a computer or another device. Currently, non-invasive BCIs transform thoughtrelated changes in the electroencephalogram (EEG) online and in real time into control signals. In such an EEG-based BCI, specific features are extracted from brain-signals, transformed into a control signal, and used to restore communication to patients with locked-in-syndrome or to control neuroprosthesis in patients with spinal cord injuries (Birbaumer et al., 1999; Pfurtscheller et al., 2008b; Wolpaw et al., 2002). In addition to these applications, which focus on communication and control, the related field of neurofeedback supports feedback training in people suffering from epilepsy, autism, stroke, and emotional or attentional disorders (Birbaumer & Cohen, 2007). Today the world of BCI applications is expanding and new fields are opening. One new direction involves BCIs to control virtual reality (VR), including BCIs for games, or using VR as a powerful feedback medium to reduce the need for BCI training (Leeb et al., 2007b; Scherer et al., 2008). Virtual environments (VE) can provide an excellent testing ground for procedures that could be adapted to real world scenarios, especially for patients with disabilities. If people can learn to control their movements or perform specific tasks in a VE, this could justify the much greater expense of building physical devices such as a wheelchair or robot arm that is controlled by a BCI. BCIs are more and more moving out of the laboratory and becoming also useful for healthy users in certain situations (Nijholt et al., 2008). One of the first efforts to combine VR and BCI technologies was Bayliss and Ballard (2000) and Bayliss (2003). They introduced a VR smart home in which users could control different appliances using a P300-based BCI. Pineda et al. (2003) showed that immersive feedback based on a computer game can help people learn to control a BCI based on imagined movement more quickly than mundane feedback, a finding we later validated with other immersive feedback (Leeb et al., 2006; 2007b). Lalor et al. (2005) used a steady-state visual evoked potential (SSVEP)-based BCI to control a character in an immersive 3-D gaming environment. Recently, Leeb et al. (2007b) have reported on exploring a smart virtual

[1]  Klaus-Robert Müller,et al.  The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects , 2007, NeuroImage.

[2]  Brendan Z. Allison,et al.  Improved signal processing approaches in an offline simulation of a hybrid brain–computer interface , 2010, Journal of Neuroscience Methods.

[3]  Xiaorong Gao,et al.  An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method , 2009, Journal of neural engineering.

[4]  Brendan Z Allison,et al.  Effects of SOA and flash pattern manipulations on ERPs, performance, and preference: implications for a BCI system. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[5]  Desney S. Tan,et al.  Brain-Computer Interfacing for Intelligent Systems , 2008, IEEE Intelligent Systems.

[6]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[7]  G Pfurtscheller,et al.  Centrally controlled heart rate changes during mental practice in immersive virtual environment: a case study with a tetraplegic. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[8]  Gernot R. Müller-Putz,et al.  Self-Paced (Asynchronous) BCI Control of a Wheelchair in Virtual Environments: A Case Study with a Tetraplegic , 2007, Comput. Intell. Neurosci..

[9]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[10]  G. Pfurtscheller,et al.  Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. , 2005, Brain research. Cognitive brain research.

[11]  H. Flor,et al.  A spelling device for the paralysed , 1999, Nature.

[12]  Dieter Schmalstieg,et al.  An Application Framework for Controlling an Avatar in a Desktop-Based Virtual Environment via a Software SSVEP BrainComputer Interface , 2010, PRESENCE: Teleoperators and Virtual Environments.

[13]  Mel Slater,et al.  An exploration of immersive virtual environments , 1995 .

[14]  Fabio Babiloni,et al.  Brain-Computer Interfaces: Towards Practical Implementations and Potential Applications , 2007, Comput. Intell. Neurosci..

[15]  E Donchin,et al.  The mental prosthesis: assessing the speed of a P300-based brain-computer interface. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[16]  Reinhold Scherer,et al.  Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components , 2005, Journal of neural engineering.

[17]  Rajesh P. N. Rao,et al.  Control of a humanoid robot by a noninvasive brain–computer interface in humans , 2008, Journal of neural engineering.

[18]  Gert Pfurtscheller,et al.  Walking from thought , 2006, Brain Research.

[19]  G. Pfurtscheller,et al.  Walking through a virtual city by thought , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[20]  G. Pfurtscheller,et al.  Brain–Computer Communication: Motivation, Aim, and Impact of Exploring a Virtual Apartment , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[21]  G. Pfurtscheller,et al.  15 years of BCI research at graz university of technology: current projects , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[22]  Horst Bischof,et al.  Toward Self-Paced Brain–Computer Communication: Navigation Through Virtual Worlds , 2008, IEEE Transactions on Biomedical Engineering.

[23]  Howard Rheingold,et al.  Virtual Reality , 1991 .

[24]  Dean J Krusienski,et al.  A comparison of classification techniques for the P300 Speller , 2006, Journal of neural engineering.

[25]  J D Bayliss,et al.  A virtual reality testbed for brain-computer interface research. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[26]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[27]  J. Wolpaw,et al.  Towards an independent brain–computer interface using steady state visual evoked potentials , 2008, Clinical Neurophysiology.

[28]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[29]  Christa Neuper,et al.  Rehabilitation with Brain-Computer Interface Systems , 2008, Computer.

[30]  John R. Smith,et al.  Steady-State VEP-Based Brain-Computer Interface Control in an Immersive 3D Gaming Environment , 2005, EURASIP J. Adv. Signal Process..

[31]  Robert Leeb.  Brain computer communication: the motivation, aim, and impact of virtual feedback , 2009 .

[32]  G. Pfurtscheller,et al.  Continuous EEG classification during motor imagery-simulation of an asynchronous BCI , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[33]  Gert Pfurtscheller,et al.  Self-paced exploration of the Austrian National Library through thought , 2007 .

[34]  Sungkil Lee,et al.  Effects of visual cues and sustained attention on spatial presence in virtual environments based on spatial and object distinction , 2008, Interact. Comput..

[35]  T. Maeshima,et al.  Autonomic response specificity during motor imagery. , 2000, Journal of physiological anthropology and applied human science.

[36]  J. Wolpaw,et al.  Patients with ALS can use sensorimotor rhythms to operate a brain-computer interface , 2005, Neurology.

[37]  F. L. D. Silva,et al.  Event-related EEG/MEG synchronization and desynchronization: basic principles , 1999, Clinical Neurophysiology.

[38]  J.D. Bayliss,et al.  Use of the evoked potential P3 component for control in a virtual apartment , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Xiaorong Gao,et al.  A BCI-based environmental controller for the motion-disabled. , 2003, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[40]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[41]  Mel Slater,et al.  Presence in immersive virtual environments , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[42]  C. Brunia,et al.  Changes in heart rate and slow brain potentials related to motor preparation and stimulus anticipation in a time estimation task. , 1987, Psychophysiology.

[43]  Brendan Z. Allison,et al.  The Hybrid BCI , 2010, Frontiers in Neuroscience.

[44]  G. Pfurtscheller,et al.  Self-Paced Operation of an SSVEP-Based Orthosis With and Without an Imagery-Based “Brain Switch:” A Feasibility Study Towards a Hybrid BCI , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[45]  J. Pineda,et al.  Learning to control brain rhythms: making a brain-computer interface possible , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[46]  M. Erb,et al.  Activation of Cortical and Cerebellar Motor Areas during Executed and Imagined Hand Movements: An fMRI Study , 1999, Journal of Cognitive Neuroscience.

[47]  Andrzej Cichocki,et al.  Fully Online Multicommand Brain-Computer Interface with Visual Neurofeedback Using SSVEP Paradigm , 2007, Comput. Intell. Neurosci..

[48]  T. Baumgartner,et al.  Virtual Reality and the Role of the Prefrontal Cortex in Adults and Children , 2009, Frontiers in neuroscience.

[49]  G. Pfurtscheller,et al.  Could the beta rebound in the EEG be suitable to realize a “brain switch”? , 2009, Clinical Neurophysiology.

[50]  M. Jeannerod,et al.  Vegetative response during imagined movement is proportional to mental effort , 1991, Behavioural Brain Research.

[51]  G. R. Muller,et al.  Brain oscillations control hand orthosis in a tetraplegic , 2000, Neuroscience Letters.

[52]  Reinhold Scherer,et al.  Combining BCI and Virtual Reality: Scouting Virtual Worlds , 2007 .

[53]  G Pfurtscheller,et al.  Toward a hybrid brain–computer interface based on imagined movement and visual attention , 2010, Journal of neural engineering.

[54]  G. Pfurtscheller,et al.  Motor imagery and action observation: Modulation of sensorimotor brain rhythms during mental control of a brain–computer interface , 2009, Clinical Neurophysiology.

[55]  L. Cohen,et al.  Brain–computer interfaces: communication and restoration of movement in paralysis , 2007, The Journal of physiology.

[56]  Mel Slater,et al.  Virtual Smart Home Controlled by Thoughts , 2009, 2009 18th IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises.

[57]  Christa Neuper,et al.  Walking by Thinking: The Brainwaves Are Crucial, Not the Muscles! , 2006, PRESENCE: Teleoperators and Virtual Environments.

[58]  G. Pfurtscheller,et al.  Dynamics of Sensorimotor Oscillations in a Motor Task , 2009 .

[59]  G Pfurtscheller,et al.  Cardiac responses induced during thought-based control of a virtual environment. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[60]  Karla Felix Navarro,et al.  A Comprehensive Survey of Brain Interface Technology Designs , 2007, Annals of Biomedical Engineering.