It Takes Nine to Smell a Rat: Neural Multi-Task Learning for Check-Worthiness Prediction

Proceedings of Recent Advances in Natural Language Processing, pages 1229–1239, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_141 1229 It Takes Nine to Smell a Rat: Neural Multi-Task Learning for Check-Worthiness Prediction Slavena Vasileva Sofia University slav.vasileva@gmail.com Pepa Atanasova University of Copenhagen pepa@di.ku.dk Llu´ıs M`arquez Amazon Core ML lluismv@amazon.com Alberto Barr´on-Cede˜no Universit`a di Bologna a.barron@unibo.it Preslav Nakov Qatar Computing Research Institute, HBKU pnakov@hbku.edu.qa Abstract We propose a multi-task deep-learning approach for estimating the check- worthiness of claims in political debates. Given a political debate, such as the 2016 US Presidential and Vice-Presidential ones, the task is to predict which state- ments in the debate should be prioritized for fact-checking. While different fact- checking organizations would naturally make different choices when analyzing the same debate, we show that it pays to learn from multiple sources simultaneously (PolitiFact, FactCheck, ABC, CNN, NPR, NYT, Chicago Tribune, The Guardian, and Washington Post) in a multi-task learning setup, even when a particular source is chosen as a target to imitate. Our evaluation shows state-of-the-art results on a standard dataset for the task of check-worthiness prediction. 1 Introduction Recent years have seen the explosion of fake news, rumors, false claims, distorted facts, half-true statements, and propaganda, which are spread- ing primarily in social media, but also via stan- dard news broadcasters. This trend became par- ticularly evident during the 2016 US Presidential campaign, which was the turning point that at- tracted wide public attention to the problem. By then, a number of organizations, e.g., FactCheck1 and Snopes2 among many others, launched fact- checking initiatives. Yet, this proved to be a very demanding manual effort, and only a relatively small number of claims could be fact-checked. Thus, it is important to prioritize what to check. 1http://www.factcheck.org/ 2http://www.snopes.com/ The task of detecting check-worthy claims has been recognized as an important stage in the pro- cess of fully automatic fact-checking. According to Vlachos and Riedel (2014) this is a multi- step process that (i) extracts statements to be fact-checked, (ii) constructs appropriate questions, (iii) obtains the answers from relevant sources, and (iv) reaches a verdict using these answers. Hassan et al. (2015a) presented a similar vision, and in a follow up work they made check-worthiness an in- tegral part of an end-to-end fact-checking system Hassan et al. (2017). Here, we approach the problem of mimick- ing the selection strategy of several renowned fact-checking organizations such as PolitiFact, FactCheck, ABC, CNN, NPR, NYT, Chicago Tri- bune, The Guardian, and The Washington Post. An important characteristic of this setup is that, per- haps due to editorial policies, fact-checking orga- nizations often select different claims for the same text, with little overlap in their choices (see Ta- bles 1 and 2). Yet, it has been previously shown that it might be beneficial to learn from the union of the selections by multiple fact-checking organi- zations (Gencheva et al., 2017). Thus, we propose a multi-task deep learning framework, in which we try to predict the choice of each and every fact- checking organization simultaneously. We …

Content truncated to 500 words