An online SSVEP-BCI system in an optical see-through augmented reality environment

OBJECTIVE This study aimed to design and evaluate a high-speed online steady-state visually evoked potential (SSVEP)-based brain-computer interface (BCI) in an optical see-through (OST) augmented reality (AR) environment. APPROACH An eight-class BCI was designed in an OST-AR headset which is wearable and allows users to see the user interface of the BCI and the device to be controlled in the same view field via the OST head-mounted display. The accuracies, information transfer rates (ITRs), and SSVEP signal characteristics of the AR-BCI were evaluated and compared with a computer screen-based BCI implemented with a laptop in offline and online cue-guided tasks. Then, the performance of the AR-BCI was evaluated in an online robotic arm control task. MAIN RESULTS The offline results obtained during the cue-guided task performed with the AR-BCI showed maximum averaged ITRs of 65.50±9.86 bits/min according to the extended canonical correlation analysis-based target identification method. The online cue-guided task achieved averaged ITRs of 65.03±11.40 bits/min. The online robotic arm control task achieved averaged ITRs of 45.57±7.40 bits/min. Compared with the screen-based BCI, some limitations of the AR environment impaired BCI performance and the quality of SSVEP signals. SIGNIFICANCE The results showed potentials of providing a high-performance brain-control interaction method by combining AR and BCI. This study could provide methodological guidelines for developing more wearable BCIs in OST-AR environments and will also encourage more interesting applications involving BCIs and AR techniques.

[1]  Santiago Rúa,et al.  Development of the supervision/control software for a multipurpose three-tank system , 2016 .

[2]  Naoki Hata,et al.  Towards Intelligent Environments: An Augmented Reality–Brain–Machine Interface Operated with a See-Through Head-Mount Display , 2011, Front. Neurosci..

[3]  Dong Ming,et al.  Incorporation of dynamic stopping strategy into the high-speed SSVEP-based BCIs , 2018, Journal of neural engineering.

[4]  Teodiano Bastos,et al.  Comparison of the influence of stimuli color on Steady-State Visual Evoked Potentials , 2015 .

[5]  Michiteru Kitazaki,et al.  AR-SSVEP for brain-machine interface: Estimating user's gaze in head-mounted display with USB camera , 2015, 2015 IEEE Virtual Reality (VR).

[6]  Yijun Wang,et al.  Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI , 2018, Int. J. Neural Syst..

[7]  Brendan Z. Allison,et al.  A feasibility study on SSVEP-based interaction with motivating and immersive virtual and augmented reality , 2017, ArXiv.

[8]  Yili Liu,et al.  A Head-Up Display-Based P300 Brain–Computer Interface for Destination Selection , 2013, IEEE Transactions on Intelligent Transportation Systems.

[9]  Chun-Yi Su,et al.  Mind Control of a Robotic Arm With Visual Fusion Technology , 2018, IEEE Transactions on Industrial Informatics.

[10]  Xiaogang Chen,et al.  Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm , 2019, Journal of neural engineering.

[11]  Xiaorong Gao,et al.  An online brain-computer interface in mobile virtual reality environments , 2019, Integr. Comput. Aided Eng..

[12]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[13]  Vicente Ferreira de Lucena,et al.  Multimodal System for Training at Distance in a Virtual or Augmented Reality Environment for Users of Electric-Powered Wheelchairs , 2016 .

[14]  Javier Minguez,et al.  A Telepresence Mobile Robot Controlled With a Noninvasive Brain–Computer Interface , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Marcia Grabowecky,et al.  Attention induces synchronization-based response gain in steady-state visual evoked potentials , 2007, Nature Neuroscience.

[16]  Kouji Takano,et al.  My thoughts through a robot's eyes: An augmented reality-brain–machine interface , 2010, Neuroscience Research.

[17]  Martin Buss,et al.  Towards robotic re-embodiment using a Brain-and-Body-Computer Interface , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Xiaogang Chen,et al.  A Benchmark Dataset for SSVEP-Based Brain–Computer Interfaces , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[19]  Antonio Frisoli,et al.  Local and Remote Cooperation With Virtual and Robotic Agents: A P300 BCI Study in Healthy and People Living With Spinal Cord Injury , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[21]  Xiaogang Chen,et al.  Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface , 2015, Journal of neural engineering.

[22]  Aiguo Song,et al.  Closed-Loop Hybrid Gaze Brain-Machine Interface Based Robotic Arm Control with Augmented Reality Feedback , 2017, Front. Neurorobot..

[23]  Hiroki Sato,et al.  Task-related component analysis for functional neuroimaging and application to near-infrared spectroscopy data , 2013, NeuroImage.

[24]  Kiyoshi Kotani,et al.  Basic Investigation of Brain–Computer Interface Combined with Augmented Reality and Development of an Improvement Method Using the Nontarget Object , 2015 .

[25]  Meng Wang,et al.  A Wearable SSVEP-Based BCI System for Quadcopter Control Using Head-Mounted Device , 2018, IEEE Access.

[26]  Henry Fuchs,et al.  Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization , 2000, Presence: Teleoperators & Virtual Environments.

[27]  Byoungho Lee,et al.  See-through optical combiner for augmented reality head-mounted display: index-matched anisotropic crystal lens , 2017, Scientific Reports.

[28]  Reinhold Scherer,et al.  Avatar navigation in virtual and augmented reality environments using an SSVEP BCI , 2010 .

[29]  Alexander Maye,et al.  Application of a single-flicker online SSVEP BCI for spatial navigation , 2017, PloS one.

[30]  S. Andersen,et al.  Effects of overt and covert attention on the steady-state visual evoked potential , 2012, Neuroscience Letters.

[31]  M. Tsolaki,et al.  EEG-Based Brain–Computer Interfaces for Communication and Rehabilitation of People with Motor Impairment: A Novel Approach of the 21st Century , 2018, Front. Hum. Neurosci..

[32]  Ekkehard Euler,et al.  Superman-like X-ray vision: Towards brain-computer interfaces for medical augmented reality , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[33]  Piotr Stawicki,et al.  Driving a Semiautonomous Mobile Robotic Car Controlled by an SSVEP-Based BCI , 2016, Comput. Intell. Neurosci..

[34]  S. Hillyard,et al.  Selective attention to stimulus location modulates the steady-state visual evoked potential. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[35]  Tonio Ball,et al.  A brain-computer interface for high-level remote control of an autonomous, reinforcement-learning-based robotic system for reaching and grasping , 2014, IUI.

[36]  Helge J. Ritter,et al.  An Augmented-Reality Based Brain-Computer Interface for Robot Control , 2010, ICONIP.

[37]  Toby P. Breckon,et al.  Using Variable Natural Environment Brain-Computer Interface Stimuli for Real-time Humanoid Robot Navigation , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[38]  Ferran Argelaguet,et al.  Brain-Computer Interfaces and Augmented Reality: a State of the Art , 2017, GBCIC.

[39]  Ronald M. Aarts,et al.  A Survey of Stimulation Methods Used in SSVEP-Based BCIs , 2010, Comput. Intell. Neurosci..

[40]  Fotis Liarokapis,et al.  BrainChat - A Collaborative Augmented Reality Brain Interface for Message Communication , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[41]  Tzyy-Ping Jung,et al.  A High-Speed Brain Speller using steady-State Visual evoked potentials , 2014, Int. J. Neural Syst..

[42]  Ferran Argelaguet,et al.  Towards BCI-Based Interfaces for Augmented Reality: Feasibility, Design and Evaluation , 2020, IEEE Transactions on Visualization and Computer Graphics.

[43]  Xu Han,et al.  A novel system of SSVEP-based human–robot coordination , 2018, Journal of neural engineering.

[44]  K. Lafleur,et al.  Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface , 2013, Journal of neural engineering.

[45]  Abderrahmane Kheddar,et al.  Navigation assistance for a BCI-controlled humanoid robot , 2014, The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent.

[46]  Bin He,et al.  Noninvasive Electroencephalogram Based Control of a Robotic Arm for Reach and Grasp Tasks , 2016, Scientific Reports.

[47]  Cuntai Guan,et al.  Online performance evaluation of motor imagery BCI with augmented-reality virtual hand feedback , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[48]  Tzyy-Ping Jung,et al.  Fast detection of covert visuospatial attention using hybrid N2pc and SSVEP features. , 2016, Journal of neural engineering.

[49]  Tzyy-Ping Jung,et al.  High-speed spelling with a noninvasive brain–computer interface , 2015, Proceedings of the National Academy of Sciences.

[50]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[51]  Yijun Wang,et al.  Enhancing Detection of SSVEPs for a High-Speed Brain Speller Using Task-Related Component Analysis , 2018, IEEE Transactions on Biomedical Engineering.