Novel image classification based on decision-level fusion of EEG and visual features

This paper presents a novel image classification based on decision-level fusion of EEG and visual features. In the proposed method, we extract the EEG features from EEG signals recorded while users stare at images, and the visual features are computed from these images. Then the classification of images is performed based on Support Vector Machine (SVM) by separately using the EEG and visual features. Furthermore, we merge the above classification results based on Supervised Learning from Multiple Experts to obtain the final classification result. This method focuses on the classification accuracy calculated from each classification result. Therefore, although classification accuracy based on EEG and visual features are different from each other, our method realizes effective integration of these classification results. In addition, we newly derive a kernelized version of the method in order to realize more accurate integration of the classification results. Consequently, our method realizes successful multimodal classification of images by the object categories that they contain.

[1]  Nobuyuki Yagi,et al.  [Survey paper] A Review of Video Retrieval Based on Image and Video Semantic Understanding , 2013 .

[2]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[3]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[4]  Gerardo Hermosillo,et al.  Supervised learning from multiple experts: whom to trust when everyone lies a bit , 2009, ICML '09.

[5]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[6]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[7]  Ryan M. Rifkin,et al.  In Defense of One-Vs-All Classification , 2004, J. Mach. Learn. Res..

[8]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[9]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[10]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[11]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[12]  Alexander A. Borbély,et al.  Sleep-deprivation: Effects on sleep and EEG in the rat , 1979, Journal of comparative physiology.

[13]  Jane Labadin,et al.  Feature selection based on mutual information , 2015, 2015 9th International Conference on IT in Asia (CITA).

[14]  Miki Haseyama,et al.  Vocal segment estimation in music pieces based on collaborative use of EEG and audio features , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[15]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[16]  Cordelia Schmid,et al.  Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[17]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[18]  Trevor Darrell,et al.  The pyramid match kernel: discriminative classification with sets of image features , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.