Self-adaptive shared control with brain state evaluation network for human-wheelchair cooperation

OBJECT For the shared control systems, how to trade off the control weight between robot autonomy and human operator is an important issue, especially for BCI-based systems. However, most of existing shared controllers have paid less attention to the effects caused by subjects with different levels of brain control ability. APPROACH In this paper, a brain state evaluation network, termed BSE-NET, is proposed to evaluate subjects' brain control ability online based on quantized attention-gated kernel reinforcement learning (QAGKRL). With the output of BSE-NET (confidence score), a shared controller is designed to dynamically adjust the control weight between robot autonomy and human operator. MAIN RESULTS The experimental results show that most of subjects achieved high and stable experimental success rate of approximately 90%. Furthermore, for subjects with different accuracy on EEG decoding, a proper confidence score can be dynamically generated to reflect their levels of brain control ability, and the proposed system can effectively adjust the control weight in all-time shared control. SIGNIFICANCE We discuss how our proposed method shows promise for BCI applications that can evaluate subjects' brain control ability online as well as provide a method for the research on self-adaptive shared control to adaptively balance control weight between subject's instruction and robot autonomy.

[1]  Mariano Sigman,et al.  Fixation-related potentials in visual search: a combined EEG and eye tracking study. , 2012, Journal of vision.

[2]  Weifeng Liu,et al.  Kernel Adaptive Filtering: A Comprehensive Introduction , 2010 .

[3]  Abdullah Akce,et al.  Remote teleoperation of an unmanned aircraft with a brain-machine interface: Theory and preliminary results , 2010, 2010 IEEE International Conference on Robotics and Automation.

[4]  Nitish V. Thakor,et al.  Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[5]  Tzyy-Ping Jung,et al.  A High-Speed Brain Speller using steady-State Visual evoked potentials , 2014, Int. J. Neural Syst..

[6]  Peng Yan,et al.  Real Time Robot Path Planning Method Based on Improved Artificial Potential Field Method , 2018, 2018 37th Chinese Control Conference (CCC).

[7]  Badong Chen,et al.  Quantized Kernel Least Mean Square Algorithm , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Benjamin Wittevrongel,et al.  Frequency- and Phase Encoded SSVEP Using Spatiotemporal Beamforming , 2016, PloS one.

[9]  Siddhartha S. Srinivasa,et al.  A policy-blending formalism for shared control , 2013, Int. J. Robotics Res..

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[11]  Thorsten O. Zander,et al.  A Survey on Unmanned Aerial Vehicle Remote Control Using Brain–Computer Interface , 2018, IEEE Transactions on Human-Machine Systems.

[12]  Tzyy-Ping Jung,et al.  High-speed spelling with a noninvasive brain–computer interface , 2015, Proceedings of the National Academy of Sciences.

[13]  Yu-Te Wang,et al.  A Comparison Study of Canonical Correlation Analysis Based Methods for Detecting Steady-State Visual Evoked Potentials , 2015, PloS one.

[14]  Ivan Volosyak,et al.  Multiple Channel Detection of Steady-State Visual Evoked Potentials for Brain-Computer Interfaces , 2007, IEEE Transactions on Biomedical Engineering.

[15]  Brice Rebsamen,et al.  A brain controlled wheelchair to navigate in familiar environments. , 2010, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[16]  Yuanqing Li,et al.  Control of a Wheelchair in an Indoor Environment Based on a Brain–Computer Interface and Automated Navigation , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[17]  Chun-Yi Su,et al.  Brain–Machine Interface and Visual Compressive Sensing-Based Teleoperation Control of an Exoskeleton Robot , 2017, IEEE Transactions on Fuzzy Systems.

[18]  Dennis J. McFarland,et al.  Brain-Computer Interface Operation of Robotic and Prosthetic Devices , 2008, Computer.

[19]  Hongqi Li,et al.  Model predictive control for a brain-controlled mobile robot , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[20]  Chun-Yi Su,et al.  Mind Control of a Robotic Arm With Visual Fusion Technology , 2018, IEEE Transactions on Industrial Informatics.

[21]  Xingang Zhao,et al.  Human Cooperative Wheelchair With Brain–Machine Interaction Based on Shared Control Strategy , 2017, IEEE/ASME Transactions on Mechatronics.

[22]  Oussama Khatib,et al.  The Potential Field Approach And Operational Space Formulation In Robot Control , 1986 .

[23]  Christian Mandel,et al.  Navigating a smart wheelchair with a brain-computer interface interpreting steady-state visual evoked potentials , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[24]  Kazuo Tanaka,et al.  Electroencephalogram-based control of an electric wheelchair , 2005, IEEE Transactions on Robotics.

[25]  M. Nuttin,et al.  A brain-actuated wheelchair: Asynchronous and non-invasive Brain–computer interfaces for continuous control of robots , 2008, Clinical Neurophysiology.

[26]  G R Müller-Putz,et al.  Toward smarter BCIs: extending BCIs through hybridization and intelligent control , 2012, Journal of neural engineering.

[27]  Xin Pan,et al.  An Intention-Driven Semi-autonomous Intelligent Robotic System for Drinking , 2017, Front. Neurorobot..

[28]  Ivan Volosyak,et al.  Impact of Frequency Selection on LCD Screens for SSVEP Based Brain-Computer Interfaces , 2009, IWANN.

[29]  Tom Carlson,et al.  Comparing shared control approaches for alternative interfaces: A wheelchair simulator experiment , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[30]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[31]  Yuanqing Li,et al.  A Hybrid Brain Computer Interface to Control the Direction and Speed of a Simulated or Real Wheelchair , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[32]  Pieter R. Roelfsema,et al.  Attention-Gated Reinforcement Learning of Internal Representations for Classification , 2005, Neural Computation.

[33]  Geoffrey A. Hollinger,et al.  Modeling user expertise for choosing levels of shared autonomy , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Kai Xu,et al.  Quantized Attention-Gated Kernel Reinforcement Learning for Brain–Machine Interface Decoding , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Kumpati S. Narendra,et al.  Adaptive and Learning Systems , 1986 .

[36]  Á. Pascual-Leone,et al.  Estimation of brain state changes associated with behavior, stimulation and epilepsy , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Eleri Cardozo,et al.  Anticipative Shared Control for Robotic Wheelchairs Used by People with Disabilities , 2015, 2015 IEEE International Conference on Autonomous Robot Systems and Competitions.

[38]  José del R. Millán,et al.  Bayesian plan recognition for Brain-Computer Interfaces , 2009, 2009 IEEE International Conference on Robotics and Automation.

[39]  Genshe Chen,et al.  Progress in EEG-Based Brain Robot Interaction Systems , 2017, Comput. Intell. Neurosci..

[40]  Jonathan R Wolpaw,et al.  Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[41]  Iñaki Iturrate,et al.  A Noninvasive Brain-Actuated Wheelchair Based on a P300 Neurophysiological Protocol and Automated Navigation , 2009, IEEE Transactions on Robotics.

[42]  Hendrik Van Brussel,et al.  User-adapted plan recognition and user-adapted shared control: A Bayesian approach to semi-autonomous wheelchair driving , 2008, Auton. Robots.

[43]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[44]  Frank Kirchner,et al.  Intrinsic interactive reinforcement learning – Using error-related potentials for real world human-robot interaction , 2017, Scientific Reports.

[45]  R. Ward,et al.  EMG and EOG artifacts in brain computer interface systems: A survey , 2007, Clinical Neurophysiology.

[46]  Hongqi Li,et al.  Model Predictive-Based Shared Control for Brain-Controlled Driving , 2020, IEEE Transactions on Intelligent Transportation Systems.

[47]  Elisa Capello,et al.  Sliding Mode Control Techniques and Artificial Potential Field for Dynamic Collision Avoidance in Rendezvous Maneuvers , 2019, IEEE Control Systems Letters.

[48]  Yili Liu,et al.  EEG-Based Brain-Controlled Mobile Robots: A Survey , 2013, IEEE Transactions on Human-Machine Systems.

[49]  Angelika Peer,et al.  Goal-recognition-based adaptive brain-computer interface for navigating immersive robotic systems , 2017, Journal of neural engineering.

[50]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[51]  Tanja Schultz,et al.  Locating user attention using eye tracking and EEG for spatio-temporal event selection , 2013, IUI '13.

[52]  Urbano Nunes,et al.  Assisted navigation based on shared-control, using discrete and sparse human-machine interfaces , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[53]  L. Cohen,et al.  Brain–computer interfaces: communication and restoration of movement in paralysis , 2007, The Journal of physiology.

[54]  Nanning Zheng,et al.  Survival kernel with application to kernel adaptive filtering , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).