A Retrospective Analysis of the Fake News Challenge Stance-Detection Task

Proceedings of the 27th International Conference on Computational Linguistics, pages 1859–1874 Santa Fe, New Mexico, USA, August 20-26, 2018. 1859 A Retrospective Analysis of the Fake News Challenge Stance Detection Task Andreas Hanselowski†, Avinesh PVS†, Benjamin Schiller†, Felix Caspelherr†, Debanjan Chaudhuri‡∗, Christian M. Meyer†, Iryna Gurevych† Research Training Group AIPHES †Computer Science Department, Technische Universität Darmstadt ‡Smart Data Analytics, University of Bonn https://www.aiphes.tu-darmstadt.de Abstract The 2017 Fake News Challenge Stage 1 (FNC-1) shared task addressed a stance classification task as a crucial first step towards detecting fake news. To date, there is no in-depth analysis paper to critically discuss FNC-1’s experimental setup, reproduce the results, and draw conclusions for next-generation stance classification methods. In this paper, we provide such an in-depth analysis for the three top-performing systems. We first find that FNC-1’s proposed evaluation metric favors the majority class, which can be easily classified, and thus overestimates the true discriminative power of the methods. Therefore, we propose a new F1-based metric yielding a changed system ranking. Next, we compare the features and architectures used, which leads to a novel feature-rich stacked LSTM model that performs on par with the best systems, but is superior in predicting minority classes. To understand the methods’ ability to generalize, we derive a new dataset and perform both in-domain and cross-domain experiments. Our qualitative and quantitative study helps interpreting the original FNC-1 scores and understand which features help improving performance and why. Our new dataset and all source code used during the reproduction study are publicly available for future research1. 1 Introduction Recently, Pomerleau and Rao (2017) organized the first Fake News Challenge2 (FNC-1) in order to foster the development of AI technology to automatically detect fake news. The challenge received much attention in the NLP community: 50 teams from both academia and industry participated. The goal of the FNC-1 challenge is to determine the perspective (or stance) of a news article relative to a given headline. An article’s stance can either agree or disagree with the headline, discuss the same topic, or it is completely unrelated. Table 1 shows four example documents illustrating these classes. Stance detection is a crucial building block for a variety of tasks, such as analyzing online debates (Walker et al., 2012; Sridhar et al., 2015; Somasundaran and Wiebe, 2010), determining the veracity of rumors on twitter (Lukasik et al., 2016; Derczynski et al., 2017), or understanding the argumentative structure of persuasive essays (Stab and Gurevych, 2017). While stance detection has been previously focused on individual sentences or phrases, the systems participating in FNC-1 have to detect the stance of an entire document, which raises many new challenges. Although the disagreeing article of Table 1 clearly leans against the headline’s claim, the fourth sentence would agree to it if considered in isolation. To properly learn from a scientific shared task, there are typically overview and analysis papers that compare the architectures, features, and results of the participating systems. To date, there is, however, no such paper for FNC-1, which is …

Content truncated to 500 words