Advancing NLP with Cognitive Language Processing Signals

When we read, our brain processes language and generates cognitive processing data such as gaze patterns and brain activity. These signals can be recorded while reading. Cognitive language processing data such as eye-tracking features have shown improvements on single NLP tasks. We analyze whether using such human features can show consistent improvement across tasks and data sources. We present an extensive investigation of the benefits and limitations of using cognitive processing data for NLP. Specifically, we use gaze and EEG features to augment models of named entity recognition, relation classification, and sentiment analysis. These methods significantly outperform the baselines and show the potential and current limitations of employing human language processing data for NLP.

[1]  Barbara Plank,et al.  What to do about non-standard (or non-canonical) language in NLP , 2016, KONVENS.

[2]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[3]  Kara D. Federmeier,et al.  Electrophysiology reveals semantic memory use in language comprehension , 2000, Trends in Cognitive Sciences.

[4]  Nora Hollenstein,et al.  ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading , 2018, Scientific Data.

[5]  Sigrid Klerke,et al.  Improving sentence compression by learning to predict gaze , 2016, NAACL.

[6]  Joachim Bingel,et al.  Sequence Classification with Human Attention , 2018, CoNLL.

[7]  Samuel Kaski,et al.  Predicting term-relevance from brain signals , 2014, SIGIR.

[8]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[9]  Bruce Wright,et al.  Thinking theta and alpha: Mechanisms of intuitive and analytical reasoning , 2019, NeuroImage.

[10]  Anders Søgaard,et al.  Learning to Predict Readability Using Eye-Movement Data From Natives and Learners , 2018, AAAI.

[11]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[12]  Joachim Bingel,et al.  Latent Multi-Task Architecture Learning , 2017, AAAI.

[13]  Robin K. Morris,et al.  Eye movements, word familiarity, and vocabulary acquisition , 2004 .

[14]  Wouter Duyck,et al.  Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading , 2017, Behavior research methods.

[15]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[16]  James Hays,et al.  WebGazer: Scalable Webcam Eye Tracking Using User Interactions , 2016, IJCAI.

[17]  Rotem Dror,et al.  Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets , 2017, TACL.

[18]  A. Jacobs,et al.  Coregistration of eye movements and EEG in natural reading: analyses and review. , 2011, Journal of experimental psychology. General.

[19]  F. Pulvermüller,et al.  Effects of word length and frequency on the human event-related potential , 2004, Clinical Neurophysiology.

[20]  Pushpak Bhattacharyya,et al.  Leveraging Cognitive Features for Sentiment Analysis , 2016, CoNLL.

[21]  Rotem Dror,et al.  The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing , 2018, ACL.

[22]  John Paulin Hansen,et al.  Low-cost gaze interaction: ready to deliver the promises , 2009, CHI Extended Abstracts.

[23]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[24]  W. Klimesch.  Alpha-band oscillations, attention, and controlled access to stored information , 2012, Trends in Cognitive Sciences.

[25]  George Panagopoulos,et al.  Multi-Task Learning for Commercial Brain Computer Interfaces , 2017, 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE).

[26]  Andrew McCallum,et al.  Integrating Probabilistic Extraction Models and Data Mining to Discover Relations and Patterns in Text , 2006, NAACL.

[27]  Shiva Taslimipoor,et al.  Using Gaze Data to Predict Multiword Expressions , 2017, RANLP.

[28]  Sabine Weiss,et al.  The contribution of EEG coherence to the investigation of language , 2003, Brain and Language.

[29]  Bao-Liang Lu,et al.  Emotion classification based on gamma-band EEG , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[30]  Pushpak Bhattacharyya,et al.  Cognitively Inspired Natural Language Processing , 2018, Cognitive Intelligence and Robotics.

[31]  Keith Rayner,et al.  Investigating the effects of a set of intercorrelated variables on eye fixation durations in reading. , 2003, Journal of experimental psychology. Learning, memory, and cognition.

[32]  Reinhold Kliegl,et al.  Synchronizing timelines: Relations between fixation durations and N400 amplitudes during sentence reading , 2007, Brain Research.

[33]  K. Rayner,et al.  Eye movements in reading words and sentences , 2007 .

[34]  Jeremy Barnes,et al.  Assessing State-of-the-Art Sentiment Models on State-of-the-Art Sentiment Datasets , 2017, WASSA@EMNLP.

[35]  Anders Søgaard,et al.  Evaluating word embeddings with fMRI and eye-tracking , 2016, RepEval@ACL.

[36]  Elena Gaudioso,et al.  Evaluation of temporal stability of eye tracking algorithms using webcams , 2016, Expert Syst. Appl..

[37]  Frank Keller,et al.  Data from eye-tracking corpora as evidence for theories of syntactic processing complexity , 2008, Cognition.

[38]  R. Shillcock,et al.  Eye Movements Reveal the On-Line Computation of Lexical Probabilities During Reading , 2003, Psychological science.

[39]  Nora Hollenstein,et al.  ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction , 2018, *SEMEVAL.

[40]  Cosima Prahm,et al.  Evaluation of consumer EEG device Emotiv EPOC , 2011 .

[41]  Joachim Bingel,et al.  Identifying beneficial task relations for multi-task learning in deep neural networks , 2017, EACL.

[42]  K. Rayner,et al.  Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity , 1986, Memory & cognition.

[43]  K. Rayner.  Visual attention in reading: Eye movements reflect cognitive processes , 1977, Memory & cognition.

[44]  Oleg V. Komogortsev,et al.  Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network , 2010, CHI Extended Abstracts.

[45]  Guillaume Lample,et al.  Neural Architectures for Named Entity Recognition , 2016, NAACL.

[46]  Nora Hollenstein,et al.  Entity Recognition at First Sight: Improving NER with Eye Movement Information , 2019, NAACL.

[47]  Anders Søgaard,et al.  Reading behavior predicts syntactic categories , 2015, CoNLL.

[48]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[49]  John Hale,et al.  Finding syntax in human encephalography with beam search , 2018, ACL.

[50]  Joachim Bingel,et al.  Weakly Supervised Part-of-speech Tagging Using Eye-tracking Data , 2016, ACL.

[51]  Joachim Bingel,et al.  Sluice networks: Learning what to share between loosely related tasks , 2017, ArXiv.

[52]  K. Rayner,et al.  Measuring word recognition in reading: eye movements and event-related potentials , 2003, Trends in Cognitive Sciences.

[53]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.