Emergent: a novel data-set for stance classification

Proceedings of NAACL-HLT 2016, pages 1163–1168, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Emergent: a novel data-set for stance classification William Ferreira Department of Computer Science University College London, UK Andreas Vlachos Department of Computer Science University of Sheffield, UK Abstract We present Emergent, a novel data-set de- rived from a digital journalism project for ru- mour debunking. The data-set contains 300 rumoured claims and 2,595 associated news articles, collected and labelled by journalists with an estimation of their veracity (true, false or unverified). Each associated article is sum- marized into a headline and labelled to indi- cate whether its stance is for, against, or ob- serving the claim, where observing indicates that the article merely repeats the claim. Thus, Emergent provides a real-world data source for a variety of natural language processing tasks in the context of fact-checking. Fur- ther to presenting the dataset, we address the task of determining the article headline stance with respect to the claim. For this purpose we use a logistic regression classifier and de- velop features that examine the headline and its agreement with the claim. The accuracy achieved was 73% which is 26% higher than the one achieved by the Excitement Open Plat- form (Magnini et al., 2014). 1 Introduction The advent of New Media, such as Twitter, Face- book, etc., enables news stories and rumours to be published in real-time to a global audience, by- passing the usual verification procedures used by more traditional Old Media news outlets. However, the line between Old and New Media is becoming blurred as news aggregators lift stories from social media and re-publish them without fact-checking. This issue could be helped by developing meth- ods for automated fact-checking of news stories, part of the reporter’s black box envisioned in Cohen et al. (2011) and one of the main objectives in com- putational journalism. While this task is related to a variety of natural language processing tasks such as textual entailment and machine comprehen- sion, it poses additional challenges due to its open- domain, real-world nature. Previous work by Vla- chos and Riedel (2014) proposed using data from fact-checking websites such as Politifact1, but the la- belling provided by the journalists is only the degree of truthfulness of the claims, without any machine- readable verdicts to supervise the various steps in deciding it. Thus, the task defined by the dataset pro- posed remains too challenging for the NLP methods currently available. In this paper we propose to use data from the Emergent Project (Silverman, 2015), a rumour de- bunking project carried out in collaboration with the Tow Center for Digital Journalism at Columbia Jour- nalism School2. Consisting of 300 claims and 2,595 associated news articles, the Emergent project con- tains a rich source of labelled data that can be used in a variety of NLP tasks, created by journalists as part of their normal workflow, thus real-world and at no annotation cost. We leverage the Emergent dataset to investigate the task of …

Content truncated to 500 words