Toward FRP-Based Brain-Machine Interfaces—Single-Trial Classification of Fixation-Related Potentials

The co-registration of eye tracking and electroencephalography provides a holistic measure of ongoing cognitive processes. Recently, fixation-related potentials have been introduced to quantify the neural activity in such bi-modal recordings. Fixation-related potentials are time-locked to fixation onsets, just like event-related potentials are locked to stimulus onsets. Compared to existing electroencephalography-based brain-machine interfaces that depend on visual stimuli, fixation-related potentials have the advantages that they can be used in free, unconstrained viewing conditions and can also be classified on a single-trial level. Thus, fixation-related potentials have the potential to allow for conceptually different brain-machine interfaces that directly interpret cortical activity related to the visual processing of specific objects. However, existing research has investigated fixation-related potentials only with very restricted and highly unnatural stimuli in simple search tasks while participant’s body movements were restricted. We present a study where we relieved many of these restrictions while retaining some control by using a gaze-contingent visual search task. In our study, participants had to find a target object out of 12 complex and everyday objects presented on a screen while the electrical activity of the brain and eye movements were recorded simultaneously. Our results show that our proposed method for the classification of fixation-related potentials can clearly discriminate between fixations on relevant, non-relevant and background areas. Furthermore, we show that our classification approach generalizes not only to different test sets from the same participant, but also across participants. These results promise to open novel avenues for exploiting fixation-related potentials in electroencephalography-based brain-machine interfaces and thus providing a novel means for intuitive human-machine interaction.

[1]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[2]  G. McConkie,et al.  The span of the effective stimulus during a fixation in reading , 1975 .

[3]  D. Navon.  Forest before trees: The precedence of global features in visual perception , 1977, Cognitive Psychology.

[4]  S Saida,et al.  Useful visual field size for pattern perception , 1979, Perception & psychophysics.

[5]  K Rayner,et al.  Reading without a fovea. , 1979, Science.

[6]  E. Donchin,et al.  Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. , 1988, Electroencephalography and clinical neurophysiology.

[7]  M. Woldorff,et al.  Distortion of ERP averages due to overlap from temporally adjacent ERPs: analysis and correction. , 2007, Psychophysiology.

[8]  M. Pomplun,et al.  Investigating the visual span in comparative search: the effects of task difficulty and divided attention , 2001, Cognition.

[9]  I. Jolliffe.  Principal Component Analysis , 2002 .

[10]  Alexander Pollatsek,et al.  The effects of frequency and predictability on eye fixations in reading: implications for the E-Z Reader model. , 2004, Journal of experimental psychology. Human perception and performance.

[11]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[12]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[13]  B. Abernethy,et al.  Probing expert anticipation with the temporal occlusion paradigm: experimental investigations of some methodological issues. , 2005, Motor control.

[14]  D. Ballard,et al.  Eye movements in natural behavior , 2005, Trends in Cognitive Sciences.

[15]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[16]  Reinhold Scherer,et al.  Steady-state visual evoked potential (SSVEP)-based communication: impact of harmonic frequency components , 2005, Journal of neural engineering.

[17]  Matthias Kaper.  P300 based brain computer interfacing , 2006 .

[18]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[19]  Ralf Engbert,et al.  Tracking the mind during reading: the influence of past, present, and future words on fixation durations. , 2006, Journal of experimental psychology. General.

[20]  J. Polich.  Updating P300: An integrative theory of P3a and P3b , 2007, Clinical Neurophysiology.

[21]  A. Lenhardt,et al.  An Adaptive P300-Based Online Brain–Computer Interface , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[22]  Sébastien Miellet,et al.  Parafoveal Magnification , 2009, Psychological science.

[23]  Yijun Wang,et al.  VEP-based brain-computer interfaces: time, frequency, and code modulations [Research Frontier] , 2009, IEEE Computational Intelligence Magazine.

[24]  Helge J. Ritter,et al.  2009 Special Issue: The MindGame: A P300-based brain-computer interface game , 2009 .

[25]  Ivan Volosyak,et al.  SSVEP-based Bremen–BCI interface—boosting information transfer rates , 2011, Journal of neural engineering.

[26]  A. Jacobs,et al.  Coregistration of eye movements and EEG in natural reading: analyses and review. , 2011, Journal of experimental psychology. General.

[27]  S. Luck,et al.  The Oxford handbook of event-related potential components , 2011 .

[28]  Chang-Hwan Im,et al.  Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard , 2012, Journal of Neuroscience Methods.

[29]  Sonja Grün,et al.  Cross-frequency interaction of the eye-movement related LFP signals in V1 of freely viewing monkeys , 2012, Front. Syst. Neurosci..

[30]  L. Vizioli,et al.  Culture modulates the temporal dynamics of global/local processing , 2013 .

[31]  Boris Reuderink,et al.  Distinguishing between target and nontarget fixations in a visual search task using fixation-related potentials. , 2013, Journal of vision.

[32]  Steven G. Luke,et al.  Co-registration of eye movements and event-related potentials in connected-text paragraph reading , 2013, Front. Syst. Neurosci..

[33]  Felix A Wichmann,et al.  When luminance increment thresholds depend on apparent lightness. , 2012, Journal of vision.

[34]  M. Arns,et al.  P300 Development across the Lifespan: A Systematic Review and Meta-Analysis , 2014, PloS one.

[35]  A. Jacobs,et al.  Co-Registration of Eye Movements and EEG in Natural Reading: Analyses & Review , 2014 .

[36]  G Müller-Putz,et al.  An independent SSVEP-based brain–computer interface in locked-in syndrome , 2014, Journal of neural engineering.

[37]  Stephanie Brandl,et al.  Robust artifactual independent component classification for BCI practitioners , 2014, Journal of neural engineering.

[38]  W. Ziegler.  The Oxford Handbook Of Event Related Potential Components , 2016 .

[39]  Helge Ritter,et al.  Using a cVEP-Based Brain-Computer Interface to Control a Virtual Agent , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[40]  Alan Kennedy,et al.  Book Review: Eye Tracking: A Comprehensive Guide to Methods and Measures , 2016, Quarterly journal of experimental psychology.