Reading Task Classification Using EEG and Eye-Tracking Data

The Zurich Cognitive Language Processing Corpus (ZuCo) provides eyetracking and EEG signals from two reading paradigms, normal reading and task-specific reading. We analyze whether machine learning methods are able to classify these two tasks using eye-tracking and EEG features. We implement models with aggregated sentence-level features as well as fine-grained word-level features. We test the models in within-subject and cross-subject evaluation scenarios. All models are tested on the ZuCo 1.0 and ZuCo 2.0 data subsets, which are characterized by differing recording procedures and thus allow for different levels of generalizability. Finally, we provide a series of control experiments to analyze the results in more detail.

[1]  Nora Hollenstein,et al.  ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading , 2018, Scientific Data.

[2]  Stefan Haufe,et al.  On the interpretation of weight vectors of linear models in multivariate neuroimaging , 2014, NeuroImage.

[3]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[4]  Joachim Bingel,et al.  Sequence Classification with Human Attention , 2018, CoNLL.

[5]  Kenneth Kreutz-Delgado,et al.  ICLabel: An automated electroencephalographic independent component classifier, dataset, and website , 2019, NeuroImage.

[6]  Noriko Tomuro,et al.  Relation Classification with Cognitive Attention Supervision , 2021, CMCL.

[7]  Uri Hasson,et al.  Keep it real: rethinking the primacy of experimental control in cognitive neuroscience , 2020, NeuroImage.

[8]  Yves Bestgen,et al.  LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach , 2021, CMCL.

[9]  Pushpak Bhattacharyya,et al.  A Survey on Using Gaze Behaviour for Natural Language Processing , 2020, IJCAI.

[10]  Nora Hollenstein,et al.  Relative Importance in Sentence Processing , 2021, ACL.

[11]  Joel Koh En Wei,et al.  Automated detection of conduct disorder and attention deficit hyperactivity disorder using decomposition and nonlinear techniques with EEG signals , 2021, Comput. Methods Programs Biomed..

[12]  Dinesh Manocha,et al.  Dynamic Graph Modeling Of Simultaneous EEG And Eye-Tracking Data For Reading Task Identification , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Kristin Lemhöfer,et al.  Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English , 2011, Behavior research methods.

[14]  Nicolas Langer,et al.  Automagic: Standardized Preprocessing of Big EEG Data , 2018 .

[15]  Tommi Kärkkäinen,et al.  Detection of developmental dyslexia with machine learning using eye movement data , 2021, Array.

[16]  Reinhold Kliegl,et al.  The cave of Shadows. Addressing the human factor with generalized additive mixed models , 2015, 1511.03120.

[17]  Nora Hollenstein,et al.  ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation , 2020, LREC.

[18]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[19]  Andrew McCallum,et al.  Integrating Probabilistic Extraction Models and Data Mining to Discover Relations and Patterns in Text , 2006, NAACL.

[20]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[21]  Udo Hahn,et al.  A Cognitive Cost Model of Annotations Based on Eye-Tracking Data , 2010, ACL.

[22]  Yong Yu,et al.  A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures , 2019, Neural Computation.

[23]  Olaf Dimigen,et al.  Unfold: an integrated toolbox for overlap correction, non-linear modeling, and regression-based EEG analysis , 2018, bioRxiv.

[24]  Ce Zhang,et al.  CogniVal: A Framework for Cognitive Word Embedding Evaluation , 2019, CoNLL.

[25]  Ping Wang,et al.  Improving Mental Task Classification by Adding High Frequency Band Information , 2008, Journal of Medical Systems.

[26]  Prospero C. Naval,et al.  Towards Learning to Read Like Humans , 2020, International Conference on Computational Collective Intelligence.

[27]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[28]  Alain de Cheveigné,et al.  ZapLine: A simple and effective method to remove power line artifacts , 2019, NeuroImage.

[29]  A. Jacobs,et al.  Coregistration of eye movements and EEG in natural reading: analyses and review. , 2011, Journal of experimental psychology. General.

[30]  Ruth E. Hogg,et al.  Individual differences in human eye movements: An oculomotor signature? , 2017, Vision Research.

[31]  Luz Rello,et al.  Detecting readers with dyslexia using machine learning with eye tracking measures , 2015, W4A.

[32]  A. Bruns.  Fourier-, Hilbert- and wavelet-based signal analysis: are they really different approaches? , 2004, Journal of Neuroscience Methods.

[33]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[34]  Takenobu Tokunaga,et al.  An Eye-tracking Study of Named Entity Annotation , 2017, RANLP.