Transfer Learning for EEG-Based Brain–Computer Interfaces: A Review of Progress Made Since 2016

A brain-computer interface (BCI) enables a user to communicate with a computer directly using brain signals. The most common non-invasive BCI modality, electroencephalogram (EEG), is sensitive to noise/artifact and suffers between-subject/within-subject non-stationarity. Therefore, it is difficult to build a generic pattern recognition model in an EEG-based BCI system that is optimal for different subjects, during different sessions, for different devices and tasks. Usually, a calibration session is needed to collect some training data for a new subject, which is time-consuming and user unfriendly. Transfer learning (TL), which utilizes data or knowledge from similar or relevant subjects/sessions/devices/tasks to facilitate learning for a new subject/session/device/task, is frequently used to reduce the amount of calibration effort. This paper reviews journal publications on TL approaches in EEG-based BCIs in the last few years, i.e., since 2016. Six paradigms and applications -- motor imagery, event-related potentials, steady-state visual evoked potentials, affective BCIs, regression problems, and adversarial attacks -- are considered. For each paradigm/application, we group the TL approaches into cross-subject/session, cross-device, and cross-task settings and review them separately. Observations and conclusions are made at the end of the paper, which may point to future research directions.

[1]  Dongrui Wu,et al.  Online driver's drowsiness estimation using domain adaptation with model fusion , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[2]  P. Ekman,et al.  Constants across cultures in the face and emotion. , 1971, Journal of personality and social psychology.

[3]  Yang Li,et al.  A Bi-Hemisphere Domain Adversarial Neural Network Model for EEG Emotion Recognition , 2021, IEEE Transactions on Affective Computing.

[4]  Chin-Teng Lin,et al.  Subject adaptation network for EEG data analysis , 2019, Appl. Soft Comput..

[5]  Yuan-Pin Lin,et al.  A subject-transfer framework for obviating inter- and intra-subject variability in EEG-based drowsiness detection , 2018, NeuroImage.

[6]  Philip S. Yu,et al.  Transfer Feature Learning with Joint Distribution Adaptation , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Dongrui Wu,et al.  Different Set Domain Adaptation for Brain-Computer Interfaces: A Label Alignment Approach , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[8]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.

[9]  Ahmed M. Azab,et al.  A review on transfer learning approaches in brain–computer interface , 2018 .

[10]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[11]  Christian Jutten,et al.  Riemannian Geometry Applied to BCI Classification , 2010, LVA/ICA.

[12]  Zhen Cui,et al.  From Regional to Global Brain: A Novel Hierarchical Spatial-Temporal Neural Network Model for EEG Emotion Recognition , 2019, IEEE Transactions on Affective Computing.

[13]  Dongrui Wu,et al.  EEG-Based Driver Drowsiness Estimation Using Feature Weighted Episodic Training , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[14]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[15]  Andrzej Cichocki,et al.  The Study of Generic Model Set for Reducing Calibration Time in P300-Based Brain–Computer Interface , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[16]  Seong-Whan Lee,et al.  Subject-Independent Brain–Computer Interfaces Based on Deep Convolutional Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[17]  E. Fetz.  Operant Conditioning of Cortical Unit Activity , 1969, Science.

[18]  Bin Hu,et al.  Exploring EEG Features in Cross-Subject Emotion Recognition , 2018, Front. Neurosci..

[19]  Dan Liu,et al.  A Fast, Efficient Domain Adaptation Technique for Cross-Domain Electroencephalography(EEG)-Based Emotion Recognition , 2017, Sensors.

[20]  J. Russell.  A circumplex model of affect. , 1980 .

[21]  Addison W. Bohannon,et al.  Spectral Transfer Learning Using Information Geometry for a User-Independent Brain-Computer Interface , 2016, Front. Neurosci..

[22]  Cuntai Guan,et al.  Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI , 2019, Journal of neural engineering.

[23]  Long Chen,et al.  Incorporation of Multiple-Days Information to Improve the Generalization of EEG-Based Emotion Recognition Over Time , 2018, Front. Hum. Neurosci..

[24]  Damien Coyle,et al.  Games, Gameplay, and BCI: The State of the Art , 2013, IEEE Transactions on Computational Intelligence and AI in Games.

[25]  Ping-Keng Jao,et al.  Improving Cross-Day EEG-Based Emotion Classification Using Robust Principal Component Analysis , 2017, Front. Comput. Neurosci..

[26]  Dongrui Wu,et al.  EEG data analysis with stacked differentiable neural computers , 2018, Neural Computing and Applications.

[27]  Tzyy-Ping Jung,et al.  Biosensor Technologies for Augmented Brain–Computer Interfaces in the Next Decades , 2012, Proceedings of the IEEE.

[28]  Todd C. Handy,et al.  Event-related potentials : a methods handbook , 2005 .

[29]  Brent Lance,et al.  Driver Drowsiness Estimation From EEG Signals Using Online Weighted Adaptation Regularization for Regression (OwARR) , 2017, IEEE Transactions on Fuzzy Systems.

[30]  Andrzej Cichocki,et al.  Modern Machine-Learning Algorithms: For Classifying Cognitive and Affective States From Electroencephalography Signals , 2020, IEEE Systems, Man, and Cybernetics Magazine.

[31]  Philip S. Yu,et al.  Adaptation Regularization: A General Framework for Transfer Learning , 2014, IEEE Transactions on Knowledge and Data Engineering.

[32]  Dongrui Wu,et al.  Collaborative Filtering for Brain-Computer Interaction Using Transfer Learning and Active Class Selection , 2013, PloS one.

[33]  Dongrui Wu,et al.  Manifold Embedded Knowledge Transfer for Brain-Computer Interfaces , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[34]  Xingcong Zhao,et al.  Multi-method Fusion of Cross-Subject Emotion Recognition Based on High-Dimensional EEG Features , 2019, Front. Comput. Neurosci..

[35]  Qisong Wang,et al.  Unsupervised domain adaptation techniques based on auto-encoder for non-stationary EEG-based emotion recognition , 2016, Comput. Biol. Medicine.

[36]  Fabien Lotte,et al.  Brain-Computer Interfaces: Beyond Medical Applications , 2012, Computer.

[37]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[38]  Albert Rothenberg,et al.  Physical Control of the Mind: Toward a Psychocivilized Society , 1970, The Yale Journal of Biology and Medicine.

[39]  Huiguang He,et al.  Multisource Transfer Learning for Cross-Subject EEG Emotion Recognition , 2020, IEEE Transactions on Cybernetics.

[40]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[41]  Wenming Zheng,et al.  EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks , 2020, IEEE Transactions on Affective Computing.

[42]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[43]  Christa Neuper,et al.  Rehabilitation with Brain-Computer Interface Systems , 2008, Computer.

[44]  Hans W. Guesgen,et al.  Small Sample Motor Imagery Classification Using Regularized Riemannian Features , 2019, IEEE Access.

[45]  Yuan-Pin Lin,et al.  Constructing a Personalized Cross-Day EEG-Based Emotion-Classification Model Using Transfer Learning , 2020, IEEE Journal of Biomedical and Health Informatics.

[46]  F. Yger,et al.  Riemannian Approaches in Brain-Computer Interfaces: A Review , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[47]  E. John,et al.  Evoked-Potential Correlates of Stimulus Uncertainty , 1965, Science.

[48]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[49]  Andrzej Cichocki,et al.  EmotionMeter: A Multimodal Framework for Recognizing Human Emotions , 2019, IEEE Transactions on Cybernetics.

[50]  Yuval Kluger,et al.  Ranking and combining multiple predictors without labeled data , 2013, Proceedings of the National Academy of Sciences.

[51]  Zhilin Zhang,et al.  Evolving Signal Processing for Brain–Computer Interfaces , 2012, Proceedings of the IEEE.

[52]  Christian Jutten,et al.  Transfer Learning: A Riemannian Geometry Framework With Applications to Brain–Computer Interfaces , 2018, IEEE Transactions on Biomedical Engineering.

[53]  Bao-Liang Lu,et al.  EEG-based vigilance estimation using extreme learning machines , 2013, Neurocomputing.

[54]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[55]  Dongrui Wu,et al.  Online and Offline Domain Adaptation for Reducing BCI Calibration Effort , 2017, IEEE Transactions on Human-Machine Systems.

[56]  Yufeng Ke,et al.  Cross-Dataset Variability Problem in EEG Decoding With Deep Learning , 2020, Frontiers in Human Neuroscience.

[57]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[58]  Charlotte Skourup,et al.  Using EEG for Mental Fatigue Assessment: A Comprehensive Look Into the Current State of the Art , 2019, IEEE Transactions on Human-Machine Systems.

[59]  Changde Du,et al.  Domain Adaptation for EEG Emotion Recognition Based on Latent Representation Similarity , 2020, IEEE Transactions on Cognitive and Developmental Systems.

[60]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[61]  Tzyy-Ping Jung,et al.  Independent Component Analysis of Electroencephalographic Data , 1995, NIPS.

[62]  Wei Liu,et al.  Vigilance Estimation Using a Wearable EOG Device in Real Driving Environment , 2020, IEEE Transactions on Intelligent Transportation Systems.

[63]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[64]  Ao Zhang,et al.  Cross-subject driver status detection from physiological signals based on hybrid feature selection and transfer learning , 2019, Expert Syst. Appl..

[65]  Zhong Yin,et al.  Cross-session classification of mental workload levels using EEG and an adaptive deep learning model , 2017, Biomed. Signal Process. Control..

[66]  Dongrui Wu,et al.  On the Vulnerability of CNN Classifiers in EEG-Based BCIs , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[67]  Ivan Volosyak,et al.  Multiple Channel Detection of Steady-State Visual Evoked Potentials for Brain-Computer Interfaces , 2007, IEEE Transactions on Biomedical Engineering.

[68]  Mahnaz Arvaneh,et al.  Weighted Transfer Learning for Improving Motor Imagery-Based Brain–Computer Interface , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[69]  Tzyy-Ping Jung,et al.  High-speed spelling with a noninvasive brain–computer interface , 2015, Proceedings of the National Academy of Sciences.

[70]  A. Nijholt,et al.  A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges , 2014 .

[71]  David Zhang,et al.  Learning Domain-Invariant Subspace Using Domain Features and Independence Maximization , 2016, IEEE Transactions on Cybernetics.

[72]  Hongzhi Qi,et al.  A Speedy Calibration Method Using Riemannian Geometry Measurement and Other-Subject Samples on A P300 Speller , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[73]  Rajesh P. N. Rao.  Brain-Computer Interfacing: An Introduction , 2010 .

[74]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[75]  Dongrui Wu,et al.  White-Box Target Attack for EEG-Based BCI Regression Problems , 2019, ICONIP.

[76]  Erhan Ekmekcioglu,et al.  Investigating the Use of Pretrained Convolutional Neural Network on Cross-Subject and Cross-Dataset EEG Emotion Recognition , 2020, Sensors.

[77]  Yuan-Pin Lin,et al.  Challenge for Affective Brain-Computer Interfaces: Non-stationary Spatio-spectral EEG Oscillations of Emotional Responses , 2019, Front. Hum. Neurosci..

[78]  Christian Jutten,et al.  Multiclass Brain–Computer Interface Classification by Riemannian Geometry , 2012, IEEE Transactions on Biomedical Engineering.

[79]  Bao-Liang Lu,et al.  Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[80]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[81]  Terrence J. Sejnowski,et al.  Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing , 2019, IEEE Transactions on Affective Computing.

[82]  Brent Lance,et al.  Switching EEG Headsets Made Easy: Reducing Offline Calibration Effort Using Active Weighted Adaptation Regularization , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[83]  F Babiloni,et al.  Passive BCI beyond the lab: current trends and future directions , 2018, Physiological measurement.

[84]  A. Mehrabian.  Basic dimensions for a general psychological theory : implications for personality, social, environmental, and developmental studies , 1980 .

[85]  Damien Coyle,et al.  A Review of Rapid Serial Visual Presentation-based Brain-1 Computer Interfaces 2 , 2017 .

[86]  T. Jung,et al.  Improving EEG-Based Emotion Classification Using Conditional Transfer Learning , 2017, Front. Hum. Neurosci..

[87]  Jun Lu,et al.  A review on transfer learning for brain-computer interface classification , 2015, 2015 5th International Conference on Information Science and Technology (ICIST).

[88]  Areej Al-Wabil,et al.  Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review , 2017 .

[89]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[90]  Ronen Talmon,et al.  Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation , 2018, IEEE Transactions on Signal Processing.

[91]  Hao Wu,et al.  A Parallel Multiscale Filter Bank Convolutional Neural Networks for Motor Imagery EEG Classification , 2019, Front. Neurosci..

[92]  Paul Sajda,et al.  Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials , 2018, Journal of neural engineering.

[93]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[94]  Philip S. Yu,et al.  Domain Invariant Transfer Kernel Learning , 2015, IEEE Transactions on Knowledge and Data Engineering.

[95]  Saeid Nahavandi,et al.  Multiclass Informative Instance Transfer Learning Framework for Motor Imagery-Based Brain-Computer Interface , 2018, Comput. Intell. Neurosci..

[96]  Dongrui Wu,et al.  Transfer Learning for Brain–Computer Interfaces: A Euclidean Space Data Alignment Approach , 2018, IEEE Transactions on Biomedical Engineering.

[97]  Francesco Piccione,et al.  User adaptive BCIs: SSVEP and P300 based interfaces , 2003, PsychNology J..

[98]  Bao-Liang Lu,et al.  A multimodal approach to estimating vigilance using EEG and forehead EOG , 2016, Journal of neural engineering.

[99]  Qiang Yang,et al.  An Overview of Multi-task Learning , 2018 .

[100]  Gernot R. Müller-Putz,et al.  Domain Adaptation Techniques for EEG-Based Emotion Recognition: A Comparative Study on Two Public Datasets , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[101]  Ye Wang,et al.  Spatial Component-wise Convolutional Network (SCCNet) for Motor-Imagery EEG Classification , 2019, 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

[102]  Bernhard Schölkopf,et al.  Transfer Learning in Brain-Computer Interfaces , 2015, IEEE Computational Intelligence Magazine.

[103]  Xiaomu Song,et al.  A study of kernel CSP-based motor imagery brain computer interface classification , 2012, 2012 IEEE Signal Processing in Medicine and Biology Symposium (SPMB).

[104]  Martial Hebert,et al.  Growing a Brain: Fine-Tuning by Increasing Model Capacity , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[105]  Dongrui Wu,et al.  EEG-Based User Reaction Time Estimation Using Riemannian Geometry Features , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[106]  J J Vidal,et al.  Toward direct brain-computer communication. , 1973, Annual review of biophysics and bioengineering.

[107]  Chun-Shu Wei,et al.  Facilitating Calibration in High-Speed BCI Spellers via Leveraging Cross-Device Shared Latent Responses , 2020, IEEE Transactions on Biomedical Engineering.

[108]  Dongrui Wu,et al.  EEG-Based Driver Drowsiness Estimation Using an Online Multi-View and Transfer TSK Fuzzy System , 2021, IEEE Transactions on Intelligent Transportation Systems.

[109]  Dennis J. McFarland,et al.  Brain–computer interfaces for communication and control , 2002, Clinical Neurophysiology.

[110]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[111]  Brent Lance,et al.  Transfer learning and active transfer learning for reducing calibration data in single-trial classification of visually-evoked potentials , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[112]  Scott E. Kerick,et al.  Brain–Computer Interface Technologies in the Coming Decades , 2012, Proceedings of the IEEE.

[113]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[114]  Bao-Liang Lu,et al.  Emotional state classification from EEG data using machine learning approach , 2014, Neurocomputing.

[115]  Guillaume Gibert,et al.  xDAWN Algorithm to Enhance Evoked Potentials: Application to Brain–Computer Interface , 2009, IEEE Transactions on Biomedical Engineering.

[116]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[117]  Dezhi Zheng,et al.  Transfer Kernel Common Spatial Patterns for Motor Imagery Brain-Computer Interface Classification , 2018, Comput. Math. Methods Medicine.

[118]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[119]  Christian Jutten,et al.  Riemannian Procrustes Analysis: Transfer Learning for Brain–Computer Interfaces , 2019, IEEE Transactions on Biomedical Engineering.