EEG-Transformer: Self-attention from Transformer Architecture for Decoding EEG of Imagined Speech

Transformers are groundbreaking architectures that have changed a flow of deep learning, and many highperformance models are developing based on transformer architectures. Transformers implemented only with attention with encoder-decoder structure following seq2seq without using RNN, but had better performance than RNN. Herein, we investigate the decoding technique for electroencephalography (EEG) composed of self-attention module from transformer architecture during imagined speech and overt speech. We performed classification of nine subjects using convolutional neural network based on EEGNet that captures temporal-spectral-spatial features from EEG of imagined speech and overt speech. Furthermore, we applied the self-attention module to decoding EEG to improve the performance and lower the number of parameters. Our results demonstrate the possibility of decoding brain activities of imagined speech and overt speech using attention modules. Also, only single channel EEG or ear-EEG can be used to decode the imagined speech for practical BCIs. Keywords—transformer, attention module, brain-computer interface, imagined speech

[1]  Paul Sajda,et al.  Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials , 2018, Journal of neural engineering.

[2]  Seong-Whan Lee,et al.  A Real-Time Movement Artifact Removal Method for Ambulatory Brain-Computer Interfaces , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[3]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[4]  Seong-Whan Lee,et al.  Network Properties in Transitions of Consciousness during Propofol-induced Sedation , 2017, Scientific Reports.

[5]  Dinggang Shen,et al.  Hybrid High-order Functional Connectivity Networks Using Resting-state Functional MRI for Mild Cognitive Impairment Diagnosis , 2017, Scientific Reports.

[6]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[7]  Klaus-Robert Müller,et al.  A lower limb exoskeleton control system based on steady state visual evoked potentials , 2015, Journal of neural engineering.

[8]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[9]  Gi-Hwan Shin,et al.  Mobile BCI dataset of scalp- and ear-EEGs with ERP and SSVEP paradigms while standing, walking, and running , 2021, Scientific Data.

[10]  Ji-Hoon Jeong,et al.  Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[11]  Panagiotis Artemiadis,et al.  Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features , 2018, Journal of neural engineering.

[12]  Daniel P Ferris,et al.  Dual-electrode motion artifact cancellation for mobile electroencephalography , 2018, Journal of neural engineering.

[13]  S. Debener,et al.  Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear , 2015, Scientific Reports.

[14]  Minji Lee,et al.  Decoding Visual Responses based on Deep Neural Networks with Ear-EEG Signals , 2020, 2020 8th International Winter Conference on Brain-Computer Interface (BCI).

[15]  Luis Villaseñor Pineda,et al.  Subjects identification using EEG-recorded imagined speech , 2019, Expert Syst. Appl..

[16]  Dinggang Shen,et al.  Strength and similarity guided group-level brain functional network construction for MCI diagnosis , 2019, Pattern Recognit..

[17]  Seong-Whan Lee,et al.  Neural Decoding of Imagined Speech and Visual Imagery as Intuitive Paradigms for BCI Communication , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[18]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[19]  Wei Guo,et al.  Classification of EEG Signals on VEP-Based BCI Systems With Broad Learning , 2021, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[20]  Yaser Jararweh,et al.  Soft Computing-Based EEG Classification by Optimal Feature Selection and Neural Networks , 2019, IEEE Transactions on Industrial Informatics.

[21]  Seong-Whan Lee,et al.  Reconstructing ERP Signals Using Generative Adversarial Networks for Mobile Brain-Machine Interface , 2020, ArXiv.

[22]  Vladlen Koltun,et al.  Exploring Self-Attention for Image Recognition , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Seong-Whan Lee,et al.  EEG Representations of Spatial and Temporal Features in Imagined Speech and Overt Speech , 2019, ACPR.

[24]  Dustin Tran,et al.  Image Transformer , 2018, ICML.

[25]  Klaus-Robert Müller,et al.  The Berlin Brain-Computer Interface (BBCI) – towards a new communication channel for online control in gaming applications , 2007, Multimedia Tools and Applications.

[26]  Stephen M. Gordon,et al.  EEGNet: A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces , 2021 .

[27]  Georg Heigold,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2021, ICLR.

[28]  Seong-Whan Lee,et al.  Subject-Independent Brain–Computer Interfaces Based on Deep Convolutional Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Ji-Hoon Jeong,et al.  Towards an EEG-based Intuitive BCI Communication System Using Imagined Speech and Visual Imagery , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[30]  Thomas Lunner,et al.  Single-channel in-ear-EEG detects the focus of auditory attention to concurrent tone streams and mixed speech , 2016, bioRxiv.

[31]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[32]  John Williamson,et al.  A High Performance Spelling System based on EEG-EOG Signals With Visual Feedback , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[33]  Thierry Dutoit,et al.  Optimizing the Performances of a P300-Based Brain–Computer Interface in Ambulatory Conditions , 2011, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[34]  Debadatta Dash,et al.  Spatial and Spectral Fingerprint in the Brain: Speaker Identification from Single Trial MEG Signals , 2019, INTERSPEECH.

[35]  Peter Hagoort,et al.  Frequency-specific directed interactions in the human brain network for language , 2017 .

[36]  Tanja Schultz,et al.  Biosignal-Based Spoken Communication: A Survey , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[37]  John Williamson,et al.  EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy , 2019, GigaScience.

[38]  Klaus-Robert Müller,et al.  A convolutional neural network for steady state visual evoked potential classification under ambulatory environment , 2017, PloS one.

[39]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[40]  Heung-Il Suk,et al.  A Novel Bayesian Framework for Discriminative Feature Extraction in Brain-Computer Interfaces , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Edward F. Chang,et al.  Speech synthesis from neural decoding of spoken sentences , 2019, Nature.