Brain-Computer Interface Control in a Virtual Reality Environment and Applications for the Internet of Things

Brain–computer interfaces (BCIs) have enabled individuals to control devices, such as spellers, robotic arms, drones, and wheelchairs, but often these BCI applications are restricted to research laboratories. With the advent of virtual reality (VR) systems and the Internet of Things (IoT) we can couple these technologies to offer real-time control of a user’s virtual and physical environment. Likewise, BCI applications are often single-use with user’s having no control outside of the restrictions placed upon the applications at the time of creation. Therefore, there is a need to create a tool that allows users the flexibility to create and modularize aspects of BCI applications for control of IoT devices and VR environments. Using a popular video game engine, Unity, and coupling it with BCI2000, we can create diverse applications that give the end-user additional autonomy during the task at hand. We demonstrate the validity of controlling a Unity-based VR environment and several commercial IoT devices via direct neural interfacing processed through BCI2000.

[1]  J D Bayliss,et al.  A virtual reality testbed for brain-computer interface research. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[2]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[3]  Karla Felix Navarro,et al.  Wearable, wireless brain computer interfaces in augmented reality environments , 2004, International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004..

[4]  John R. Smith,et al.  Steady-State VEP-Based Brain-Computer Interface Control in an Immersive 3D Gaming Environment , 2005, EURASIP J. Adv. Signal Process..

[5]  Bin He,et al.  BRAIN^COMPUTER INTERFACE , 2007 .

[6]  Gernot R. Müller-Putz,et al.  Self-Paced (Asynchronous) BCI Control of a Wheelchair in Virtual Environments: A Case Study with a Tetraplegic , 2007, Comput. Intell. Neurosci..

[7]  Reinhold Scherer,et al.  Combining BCI and Virtual Reality: Scouting Virtual Worlds , 2007 .

[8]  Bin He,et al.  Cortical Imaging of Event-Related (de)Synchronization During Online Control of Brain-Computer Interface Using Minimum-Norm Estimates in Frequency Domain , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[9]  Michitaka Hirose,et al.  Brain-Computer Interfaces, Virtual Reality, and Videogames , 2008, Computer.

[10]  Ricardo Ron-Angevin,et al.  Brain–computer interface: Changes in performance using virtual reality techniques , 2009, Neuroscience Letters.

[11]  O. Arias-Carrión,et al.  EEG-based Brain-Computer Interfaces: An Overview of Basic Concepts and Clinical Applications in Neurorehabilitation , 2010, Reviews in the neurosciences.

[12]  C. Neuper,et al.  Combining Brain–Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges , 2010, Front. Neurosci..

[13]  Bin He,et al.  EEG Control of a Virtual Helicopter in 3-Dimensional Space Using Intelligent Control Strategies , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[14]  Dieter Schmalstieg,et al.  An Application Framework for Controlling an Avatar in a Desktop-Based Virtual Environment via a Software SSVEP BrainComputer Interface , 2010, PRESENCE: Teleoperators and Virtual Environments.

[15]  Mel Slater,et al.  The Sense of Embodiment in Virtual Reality , 2012, PRESENCE: Teleoperators and Virtual Environments.

[16]  G. Edlinger,et al.  A dry electrode concept for SMR, P300 and SSVEP based BCIs , 2012, 2012 ICME International Conference on Complex Medical Engineering (CME).

[17]  S. Kim,et al.  Mirror Therapy for Phantom Limb Pain , 2012, The Korean journal of pain.

[18]  Tzyy-Ping Jung,et al.  Real-time modeling and 3D visualization of source dynamics and connectivity using wearable EEG , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  Camarin E. Rolle,et al.  Video game training enhances cognitive control in older adults , 2013, Nature.

[20]  Bin He,et al.  Brain–Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives , 2014, IEEE Transactions on Biomedical Engineering.

[21]  Bin He,et al.  Noninvasive Brain-Computer Interfaces Based on Sensorimotor Rhythms , 2015, Proceedings of the IEEE.

[22]  Bin He,et al.  EEG Source Imaging Enhances the Decoding of Complex Right-Hand Motor Imagery Tasks , 2016, IEEE Transactions on Biomedical Engineering.

[23]  Bin He,et al.  Noninvasive Electroencephalogram Based Control of a Robotic Arm for Reach and Grasp Tasks , 2016, Scientific Reports.

[24]  Kup-Sze Choi,et al.  Evaluation of Motor Training Performance in 3D Virtual Environment via Combining Brain-computer Interface and Haptic Feedback , 2017 .