Toward Imagined Speech based Smart Communication System: Potential Applications on Metaverse Conditions

Recently, brain-computer interface (BCI) system is aiming to provide user-friendly and intuitive means of communication. Imagined speech has become an alternative neuroparadigm for communicative BCI since it relies directly on a person’s speech production process, rather than using speechunrelated neural activity as the method of communication. Together with the current trends on BCI, we suggest a brainto-speech (BTS) system that operates by real-time decoding of multi-class imagined speech electroencephalography (EEG). Using the thirteen-class imagined speech data of nine subjects, we performed pseudo-online analysis in order to investigate the potential use of virtual BTS system in the real-world. Average accuracy of 46.54 ± 9.37 % (chance level = 7.7 %) and 75.56 ± 5.83 % (chance level = 50 %) was acquired in the thirteen-class and binary pseudo-online analysis, respectively. Together with the pseudo-online analysis of imagined speech decoding, we suggest possible form of future applications of imagined speech BCI as means of intuitive BCI communication. We provide potential applications on virtual smart home system and virtual assistant controlled by imagined speech. The virtual BTS system in this paper displays the possible form of real-world application and online training platform.

[1]  Dinggang Shen,et al.  Strength and similarity guided group-level brain functional network construction for MCI diagnosis , 2019, Pattern Recognit..

[2]  Klaus-Robert Müller,et al.  The Berlin Brain-Computer Interface (BBCI) – towards a new communication channel for online control in gaming applications , 2007, Multimedia Tools and Applications.

[3]  Siamac Fazli,et al.  Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI , 2015, Pattern Recognit..

[4]  Luis Alfredo Moctezuma,et al.  Multi-objective optimization for EEG channel selection and accurate intruder detection in an EEG-based subject identification system , 2020, Scientific Reports.

[5]  John Williamson,et al.  A High Performance Spelling System based on EEG-EOG Signals With Visual Feedback , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[6]  John Williamson,et al.  EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy , 2019, GigaScience.

[7]  Stephen M. Gordon,et al.  EEGNet: A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces , 2021 .

[8]  Heung-Il Suk,et al.  Subject and class specific frequency bands selection for multiclass motor imagery classification , 2011, Int. J. Imaging Syst. Technol..

[9]  Seong-Whan Lee,et al.  Subject-Independent Brain–Computer Interfaces Based on Deep Convolutional Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Ji-Hoon Jeong,et al.  Towards an EEG-based Intuitive BCI Communication System Using Imagined Speech and Visual Imagery , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[11]  Seong-Whan Lee,et al.  A Real-Time Movement Artifact Removal Method for Ambulatory Brain-Computer Interfaces , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[12]  Ji-Hoon Jeong,et al.  Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[13]  Yaser Jararweh,et al.  Soft Computing-Based EEG Classification by Optimal Feature Selection and Neural Networks , 2019, IEEE Transactions on Industrial Informatics.

[14]  Liina Pylkkänen,et al.  The neural basis of combinatory syntax and semantics , 2019, Science.

[15]  Panagiotis Artemiadis,et al.  Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features , 2018, Journal of neural engineering.

[16]  Christian Brodbeck,et al.  Neural source dynamics of brain responses to continuous stimuli: Speech processing from acoustics to comprehension , 2017, NeuroImage.

[17]  Dinggang Shen,et al.  Hybrid High-order Functional Connectivity Networks Using Resting-state Functional MRI for Mild Cognitive Impairment Diagnosis , 2017, Scientific Reports.

[18]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[19]  Bahar Khalighinejad,et al.  Towards reconstructing intelligible speech from the human auditory cortex , 2019, Scientific Reports.

[20]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[21]  Seong-Whan Lee,et al.  Error Correction Regression Framework for Enhancing the Decoding Accuracies of Ear-EEG Brain–Computer Interfaces , 2020, IEEE Transactions on Cybernetics.

[22]  Anna Gawlinski,et al.  Communication boards in critical care: patients' views. , 2006, Applied nursing research : ANR.

[23]  Quentin Barthélemy,et al.  The Riemannian Potato Field: A Tool for Online Signal Quality Index of EEG , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[24]  Seong-Whan Lee,et al.  Audio Dequantization for High Fidelity Audio Generation in Flow-based Neural Vocoder , 2020, INTERSPEECH.

[25]  Seong-Whan Lee,et al.  Neural Decoding of Imagined Speech and Visual Imagery as Intuitive Paradigms for BCI Communication , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  John Williamson,et al.  Mental fatigue in central-field and peripheral-field steady-state visually evoked potential and its effects on event-related potential responses , 2018, Neuroreport.

[27]  Minji Lee,et al.  Decoding Visual Responses based on Deep Neural Networks with Ear-EEG Signals , 2020, 2020 8th International Winter Conference on Brain-Computer Interface (BCI).

[28]  Seong-Whan Lee,et al.  EEG Representations of Spatial and Temporal Features in Imagined Speech and Overt Speech , 2019, ACPR.

[29]  Paul Sajda,et al.  Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials , 2018, Journal of neural engineering.

[30]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[31]  Edward F. Chang,et al.  Speech synthesis from neural decoding of spoken sentences , 2019, Nature.