A Time-Frequency Distribution-Based Approach for Decoding Visually Imagined Objects Using EEG Signals

This paper investigates the possibility of using visual imagery tasks, which are mental imagery tasks that involve imagining the images of objects perceptually without seeing them, as a control paradigm that can increase the control’s dimensionality of electroencephalography (EEG)-based brain-computer interfaces. Specifically, we propose an EEG-based approach for decoding visually imagined objects by using the Choi-Williams time-frequency distribution to analyze the EEG signals in the joint time-frequency domain and extract a set of twelve time-frequency features (TFFs). The extracted TFFs are used to construct a multi-class support vector machine classifier to decode various visually imagined objects. To validate the performance of our proposed approach, we have recorded an EEG dataset for 16 healthy subjects while imagining objects that belong to four different categories, namely nature (fruits and animals), decimal digits, English alphabet (capital letters), and arrow shapes (arrows with different colors and orientations). Moreover, we have designed two performance evaluation analyses, namely the channel-based analysis and feature-based analysis, to quantify the impact of utilizing different groups of EEG electrodes that cover various regions on the scalp and the effect of reducing the dimensionality of the extracted TFFs on the performance of our proposed approach in decoding the imagined objects within each of the four categories. The experimental results demonstrate the efficacy of our proposed approach in decoding visually imagined objects. Particularly, the average decoding accuracies obtained for each of the four categories were as high as 96.67%, 93.64%, 88.95%, and 92.68%.

[1]  Mohammad I. Daoud,et al.  EEG-Based Emotion Recognition Using Quadratic Time-Frequency Distribution , 2018, Sensors.

[2]  J. Wolpaw,et al.  Brain-computer communication: unlocking the locked in. , 2001, Psychological bulletin.

[3]  V. Vapnik.  The Support Vector Method of Function Estimation , 1998 .

[4]  Brent Lance,et al.  EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces , 2016, Journal of neural engineering.

[5]  K. Lafleur,et al.  Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface , 2013, Journal of neural engineering.

[6]  U. Rajendra Acharya,et al.  Application of entropies for automated diagnosis of epilepsy using EEG signals: A review , 2015, Knowl. Based Syst..

[7]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[8]  M. Jeannerod.  Mental imagery in the motor context , 1995, Neuropsychologia.

[9]  S. Kosslyn,et al.  Brain areas underlying visual mental imagery and visual perception: an fMRI study. , 2004, Brain research. Cognitive brain research.

[10]  I. Toni,et al.  Shared Representations for Working Memory and Mental Imagery in Early Visual Cortex , 2013, Current Biology.

[11]  William J. Williams,et al.  Improved time-frequency representation of multicomponent signals using exponential kernels , 1989, IEEE Trans. Acoust. Speech Signal Process..

[12]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[13]  Dong Wen,et al.  Review of Sparse Representation-Based Classification Methods on EEG Signal Processing for Epilepsy Detection, Brain-Computer Interface and Cognitive Impairment , 2016, Front. Aging Neurosci..

[14]  Carlo Menon,et al.  EEG Classification of Different Imaginary Movements within the Same Limb , 2015, PloS one.

[15]  Mohammad I. Daoud,et al.  EEG-based BCI system for decoding finger movements within the same hand , 2019, Neuroscience Letters.

[16]  Jaime Gómez Gil,et al.  Brain Computer Interfaces, a Review , 2012, Sensors.

[17]  Nigel J. T. Thomas.  Mental Imagery, Philosophical Issues about , 2006 .

[18]  M. Brodeur,et al.  The Bank of Standardized Stimuli (BOSS), a New Set of 480 Normative Photos of Objects to Be Used as Visual Stimuli in Cognitive Research , 2010, PloS one.

[19]  Andrzej Cichocki,et al.  Correlation-based channel selection and regularized feature optimization for MI-based BCI , 2019, Neural Networks.

[20]  J. O. Toole.  Discrete quadratic time-frequency distributions: Definition, computation, and a newborn electroencephalogram application , 2009 .

[21]  Mohammad Reza Daliri,et al.  Decoding Objects of Basic Categories from Electroencephalographic Signals Using Wavelet Transform and Support Vector Machines , 2014, Brain Topography.

[22]  Gert Pfurtscheller,et al.  Motor imagery and direct brain-computer communication , 2001, Proc. IEEE.

[23]  G. R. Muller,et al.  Brain oscillations control hand orthosis in a tetraplegic , 2000, Neuroscience Letters.

[24]  Klaus-Robert Müller,et al.  Neurophysiological predictor of SMR-based BCI performance , 2010, NeuroImage.

[25]  B. Gulyás,et al.  Visual memory, visual imagery, and visual recognition of large field patterns by the human brain: functional anatomy by positron emission tomography. , 1995, Cerebral cortex.

[26]  U. Rajendra Acharya,et al.  Deep learning for healthcare applications based on physiological signals: A review , 2018, Comput. Methods Programs Biomed..

[27]  Robert Oostenveld,et al.  Identifying Object Categories from Event-Related EEG: Toward Decoding of Conceptual Representations , 2010, PloS one.

[28]  Sung Chan Jun,et al.  High Theta and Low Alpha Powers May Be Indicative of BCI-Illiteracy in Motor Imagery , 2013, PloS one.

[29]  Minkyu Ahn,et al.  Journal of Neuroscience Methods , 2015 .

[30]  G. Pfurtscheller,et al.  Conversion of EEG activity into cursor movement by a brain-computer interface (BCI) , 2004, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[31]  Ehsan Tarkesh Esfahani,et al.  Classification of primitive shapes using brain-computer interfaces , 2012, Comput. Aided Des..

[32]  S. Kosslyn,et al.  Visual memory and visual mental imagery recruit common control and sensory regions of the brain , 2012, Cognitive neuroscience.

[33]  Carlo Menon,et al.  Classifying three imaginary states of the same upper extremity using time-domain features , 2017, PloS one.

[34]  N. Tzourio,et al.  A Positron Emission Tomography Study of Visual and Mental Spatial Exploration , 1995, Journal of Cognitive Neuroscience.

[35]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[36]  A. Doud,et al.  Continuous Three-Dimensional Control of a Virtual Helicopter Using a Motor Imagery Based Brain-Computer Interface , 2011, PloS one.

[37]  Jussi T. Lindgren,et al.  Attending to Visual Stimuli versus Performing Visual Imagery as a Control Strategy for EEG-based Brain-Computer Interfaces , 2018, Scientific Reports.

[38]  Ulrich H.-G. Kreßel,et al.  Pairwise classification and support vector machines , 1999 .

[39]  Clemens Brunner,et al.  Spatial filtering and selection of optimized components in four class motor imagery EEG data using independent components analysis , 2007, Pattern Recognit. Lett..

[40]  Tomoyasu Horikawa,et al.  Generic decoding of seen and imagined objects using hierarchical visual features , 2015, Nature Communications.

[41]  L. Cohen,et al.  Time-frequency distributions-a review , 1989, Proc. IEEE.

[42]  Yuguo Yu,et al.  Decoding English Alphabet Letters Using EEG Phase Information , 2018, Front. Neurosci..

[43]  Sadasivan Puthusserypady,et al.  An end-to-end deep learning approach to MI-EEG signal classification for BCIs , 2018, Expert Syst. Appl..

[44]  Tomoyasu Horikawa,et al.  Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features , 2016, Front. Comput. Neurosci..

[45]  A. Frolov,et al.  Brain-Computer Interface Based on Generation of Visual Images , 2011, PloS one.

[46]  Mohammad I. Daoud,et al.  A Deep Learning Framework for Decoding Motor Imagery Tasks of the Same Hand Using EEG Signals , 2019, IEEE Access.

[47]  Ugur Halici,et al.  A novel deep learning approach for classification of EEG motor imagery signals , 2017, Journal of neural engineering.

[48]  Lina Yao,et al.  A Survey on Deep Learning based Brain Computer Interface: Recent Advances and New Frontiers , 2019, ArXiv.

[49]  W. De Clercq,et al.  Automatic Removal of Ocular Artifacts in the EEG without an EOG Reference Channel , 2006, Proceedings of the 7th Nordic Signal Processing Symposium - NORSIG 2006.

[50]  Xingyu Wang,et al.  Optimized Motor Imagery Paradigm Based on Imagining Chinese Characters Writing Movement , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[51]  P. Sajda,et al.  Temporal characterization of the neural correlates of perceptual decision making in the human brain. , 2006, Cerebral cortex.

[52]  S. Kosslyn,et al.  Mental Imagery: Functional Mechanisms and Clinical Applications , 2015, Trends in Cognitive Sciences.

[53]  Ke Liao,et al.  Decoding Individual Finger Movements from One Hand Using Human EEG Signals , 2014, PloS one.

[54]  Mohammad I. Daoud,et al.  EEG-Based Brain-Computer Interface for Decoding Motor Imagery Tasks within the Same Hand Using Choi-Williams Time-Frequency Distribution , 2017, Sensors.

[55]  Paolo Castiglioni.  Choi–Williams Distribution , 2005 .

[56]  Fraser Milton,et al.  The neural correlates of visual imagery vividness – An fMRI study and literature review , 2017, Cortex.

[57]  Bin He,et al.  EEG Control of a Virtual Helicopter in 3-Dimensional Space Using Intelligent Control Strategies , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[58]  Klaus-Robert Müller,et al.  Eyes-closed hybrid brain-computer interface employing frontal brain activation , 2018, PloS one.

[59]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[61]  Mohammad I. Daoud,et al.  Tonic Cold Pain Detection Using Choi–Williams Time-Frequency Distribution Analysis of EEG Signals: A Feasibility Study , 2019, Applied Sciences.

[62]  Yu Zhang,et al.  EEG classification using sparse Bayesian extreme learning machine for brain–computer interface , 2018, Neural Computing and Applications.

[63]  Leontios J. Hadjileontiadis,et al.  EEG-Based Tonic Cold Pain Characterization Using Wavelet Higher Order Spectral Features , 2015, IEEE Transactions on Biomedical Engineering.

[64]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[65]  Wolfram Burgard,et al.  Deep learning with convolutional neural networks for EEG decoding and visualization , 2017, Human brain mapping.

[66]  Mohammad I. Daoud,et al.  EEG-based tonic cold pain recognition system using wavelet transform , 2017, Neural Computing and Applications.

[67]  Zhiquan Wang,et al.  Recognition of human activities using SVM multi-class classifier , 2010, Pattern Recognit. Lett..

[68]  Boualem Boashash,et al.  Time-Frequency Signal Analysis and Processing: A Comprehensive Reference , 2015 .

[69]  Boualem Boashash,et al.  Automatic signal abnormality detection using time-frequency features and machine learning: A newborn EEG seizure case study , 2016, Knowl. Based Syst..

[70]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[71]  Michael X Cohen,et al.  Analyzing Neural Time Series Data: Theory and Practice , 2014 .

[72]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[73]  Francisco Sepulveda,et al.  Classifying mental tasks based on features of higher-order statistics from EEG signals in brain-computer interface , 2008, Inf. Sci..

[74]  S. Kosslyn,et al.  Neural foundations of imagery , 2001, Nature Reviews Neuroscience.

[75]  M. Farah,et al.  A functional MRI study of mental image generation , 1997, Neuropsychologia.