MOABB: trustworthy algorithm benchmarking for BCIs

OBJECTIVE Brain-computer interface (BCI) algorithm development has long been hampered by two major issues: small sample sets and a lack of reproducibility. We offer a solution to both of these problems via a software suite that streamlines both the issues of finding and preprocessing data in a reliable manner, as well as that of using a consistent interface for machine learning methods. APPROACH By building on recent advances in software for signal analysis implemented in the MNE toolkit, and the unified framework for machine learning offered by the scikit-learn project, we offer a system that can improve BCI algorithm development. This system is fully open-source under the BSD licence and available at https://github.com/NeuroTechX/moabb. MAIN RESULTS We analyze a set of state-of-the-art decoding algorithms across 12 open access datasets, including over 250 subjects. Our results show that even for the best methods, there are datasets which do not show significant improvements, and further that many previously validated methods do not generalize well outside the datasets they were tested on. SIGNIFICANCE Our analysis confirms that BCI algorithms validated on single datasets are not representative, highlighting the need for more robust validation in the machine learning for BCIs community.

[1]  G. Pfurtscheller,et al.  Brain–Computer Communication: Motivation, Aim, and Impact of Exploring a Virtual Apartment , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[2]  Bangyan Zhou,et al.  A Fully Automated Trial Selection Method for Optimization of Motor Imagery Based Brain-Computer Interface , 2016, PloS one.

[3]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[4]  Martin Luessi,et al.  MNE software for processing MEG and EEG data , 2014, NeuroImage.

[5]  Louis B. Rall,et al.  Automatic differentiation , 1981 .

[6]  Scott Makeig,et al.  BCILAB: a platform for brain–computer interface development , 2013, Journal of neural engineering.

[7]  Cuntai Guan,et al.  Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms , 2011, IEEE Transactions on Biomedical Engineering.

[8]  B. Allison,et al.  BCI Demographics: How Many (and What Kinds of) People Can Use an SSVEP BCI? , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[9]  A. S. Rodionov,et al.  Comparison of linear, nonlinear and feature selection methods for EEG signal classification , 2004, International Conference on Actual Problems of Electron Devices Engineering, 2004. APEDE 2004..

[10]  G. Pfurtscheller,et al.  The BCI competition III: validating alternative approaches to actual BCI problems , 2006, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[11]  E. Suchman,et al.  The American Soldier: Adjustment During Army Life. , 1949 .

[12]  Jichai Jeong,et al.  Dataset for EEG + NIRS Single-Trial Classification , 2022 .

[13]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[14]  Christian Jutten,et al.  Classification of covariance matrices using a Riemannian-based kernel for BCI applications , 2013, Neurocomputing.

[15]  Guillaume Gibert,et al.  OpenViBE: An Open-Source Software Platform to Design, Test, and Use BrainComputer Interfaces in Real and Virtual Environments , 2010, PRESENCE: Teleoperators and Virtual Environments.

[16]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[17]  Alfred O. Hero,et al.  Shrinkage Algorithms for MMSE Covariance Estimation , 2009, IEEE Transactions on Signal Processing.

[18]  L. Kirkup,et al.  A direct comparison of wet, dry and insulating bioelectric recording electrodes. , 2000, Physiological measurement.

[19]  Daniel Sánchez Morillo,et al.  Dry EEG Electrodes , 2014, Sensors.

[20]  Dong Ming,et al.  Evaluation of EEG Oscillatory Patterns and Cognitive Process during Simple and Compound Limb Motor Imagery , 2014, PloS one.

[21]  Klaus-Robert Müller,et al.  The non-invasive Berlin Brain–Computer Interface: Fast acquisition of effective performance in untrained subjects , 2007, NeuroImage.

[22]  Bin He,et al.  Brain–Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives , 2014, IEEE Transactions on Biomedical Engineering.

[23]  Gernot R. Müller-Putz,et al.  Individually Adapted Imagery Improves Brain-Computer Interface Performance in End-Users with Disability , 2015, PloS one.

[24]  Gernot R. Müller-Putz,et al.  Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier , 2016, Biomedizinische Technik. Biomedical engineering.

[25]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[26]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[27]  Alexandre Barachant,et al.  Commande robuste d'un effecteur par une interface cerveau machine EEG asynchrone , 2012 .

[28]  A. Kadish,et al.  Uncertainty principle of signal-averaged electrocardiography. , 2000, Circulation.

[29]  Moritz Grosse-Wentrup,et al.  Beamforming in Noninvasive Brain–Computer Interfaces , 2009, IEEE Transactions on Biomedical Engineering.

[30]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[31]  Michael S. Lazar,et al.  Spatial patterns underlying population differences in the background EEG , 2005, Brain Topography.

[32]  E. Suchman,et al.  The American soldier: Adjustment during army life. (Studies in social psychology in World War II), Vol. 1 , 1949 .

[33]  Ad Aertsen,et al.  Review of the BCI Competition IV , 2012, Front. Neurosci..

[34]  Sung Chan Jun,et al.  EEG datasets for motor imagery brain–computer interface , 2017, GigaScience.

[35]  Bernhard Schölkopf,et al.  Transfer Learning in Brain-Computer Interfaces , 2015, IEEE Computational Intelligence Magazine.

[36]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[37]  Christa Neuper,et al.  Autocalibration and Recurrent Adaptation: Towards a Plug and Play Online ERD-BCI , 2012, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[38]  Martin Luessi,et al.  MEG and EEG data analysis with MNE-Python , 2013, Front. Neuroinform..