Factuality Checking in News Headlines with Eye Tracking

We study whether it is possible to infer if a news headline is true or false using only the movement of the human eyes when reading news headlines. Our study with 55 participants who are eye-tracked when reading 108 news headlines (72 true, 36 false) shows that false headlines receive statistically significantly less visual attention than true headlines. We further build an ensemble learner that predicts news headline factuality using only eye-tracking measurements. Our model yields a mean AUC of 0.688 and is better at detecting false than true headlines. Through a model analysis, we find that eye-tracking 25 users when reading 3-6 headlines is sufficient for our ensemble learner.

[1]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[2]  John Shawe-Taylor,et al.  Can eyes reveal interest? Implicit queries from gaze patterns , 2009, User Modeling and User-Adapted Interaction.

[3]  Jacek Gwizdka.  News stories relevance effects on eye-movements , 2014, ETRA.

[4]  Verónica Pérez-Rosas,et al.  Automatic Detection of Fake News , 2017, COLING.

[5]  Peter Brusilovsky,et al.  Inferring word relevance from eye-movements of readers , 2011, IUI '11.

[6]  Samuel Kaski,et al.  Learning to learn implicit queries from gaze patterns , 2008, ICML '08.

[7]  Edward Cutrell,et al.  What are you looking for?: an eye-tracking study of information usage in web search , 2007, CHI.

[8]  Andreas Dengel,et al.  Query expansion using gaze-based feedback on the subdocument level , 2008, SIGIR '08.

[9]  Jacek Gwizdka,et al.  Relating eye-tracking measures with changes in knowledge on search tasks , 2018, ETRA.

[10]  William Yang Wang.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection , 2017, ACL.

[11]  Rachel Rudinger,et al.  Hypothesis Only Baselines in Natural Language Inference , 2018, *SEMEVAL.

[12]  Stephan Winter,et al.  Selective attention in the news feed: An eye-tracking study on the perception and selection of political news posts on Facebook , 2018, New Media Soc..

[13]  Lang Wu,et al.  Mixed Effects Models for Complex Data , 2019 .

[14]  Andreas Dengel,et al.  Attentive documents: Eye tracking as implicit feedback for information retrieval and beyond , 2012, TIIS.

[15]  John Shawe-Taylor,et al.  Information Retrieval by Inferring Implicit Queries from Eye Movements , 2007, AISTATS.