Stance Detection for the Fake News Challenge with Attention and Conditional Encoding

The in-progress Fake News Challenge is a public challenge tasking competitors to develop a stance detection tool that could ultimately be incorporated into a larger automatic fact-checking pipeline. 49,972 body-headline pairs are labeled with either ”Unrelated”, ”Discusses”, ”Agrees”, or ”Disagrees”, and it is the goal of the stance detection task to predict these labels. We applied the concepts of neural attention and conditional encoding to long short-term memory networks (LSTM) ultimately achieving a preliminary competition score of 0.808, improving over the competition baseline of 0.795 that relies on several hand-crafted linguistic features. Four models were evaluated: Bag of Words (BOW), basic LSTM, LSTM with attention, conditional encoding LSTM with attention (CEA LSTM). The attention models outperformed the simpler models on all performance metrics on the test set. In particular, the models with neural attention were able to achieve significantly higher F1 scores predicting the infrequent stances ”Agrees” and ”Disagrees”.