STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4086–4101 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4086 STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval Costanza Conforti1, Jakob Berndt2, Mohammad Taher Pilehvar1,3, Chryssi Giannitsarou2, Flavio Toxvaerd2, Nigel Collier1 1 Language Technology Lab, University of Cambridge 2 Faculty of Economics, University of Cambridge 3 Tehran Institute for Advanced Studies, Iran {cc918,jb2088}@cam.ac.uk Abstract We present a new challenging news dataset that targets both stance detection (SD) and fine- grained evidence retrieval (ER). With its 3,291 expert-annotated articles, the dataset consti- tutes a high-quality benchmark for future re- search in SD and multi-task learning. We pro- vide a detailed description of the corpus collec- tion methodology and carry out an extensive analysis on the sources of disagreement be- tween annotators, observing a correlation be- tween their disagreement and the diffusion of uncertainty around a target in the real world. Our experiments show that the dataset poses a strong challenge to recent state-of-the-art mod- els. Notably, our dataset aligns with an exist- ing Twitter SD dataset: their union thus ad- dresses a key shortcoming of previous work, by providing the first dedicated resource to study multi-genre SD as well as the interplay of signals from social media and news sources in rumour verification. 1 Introduction Starting from early work by Agrawal et al. (2003), Stance Detection (SD) has gained increasing inter- est from the research community (Zubiaga et al., 2018a). Recent work in SD has mostly focused on modeling user-generated data (Mohammad et al., 2017; K¨uc¸¨uk and Can, 2020). However, SD on complex and articulated texts, such as news arti- cles, has been considerably less studied, mainly due to the scarcity of published datasets (Pomerleau and Rao, 2017; Hanselowski et al., 2019). More- over, research on user-generated SD and news SD has proceeded on parallel and independent tracks, neglecting the deep mutual influence that exists between social media and news sources (Canter, 2015; Kostkova et al., 2017). In this paper, we seek to fill this gap, introduc- ing STANDER (STANce Detection & Evidence Re- trieval), a new expert-annotated dataset which is labeled for both news SD and fine-grained ER. STANDER collects news articles in English from high-reputation sources which discuss four recent mergers and acquisitions (M&A) operations be- tween major healthcare companies in the US (Ta- ble 1). The term M&A refers to the process by which the ownership of a company (the tar- get) is transferred to another company (the buyer). An M&A process (merger) ranges from informal talks between the companies to the closing of the deal; high secrecy is involved and discussions are usually not publicly disclosed during its early stages (Bruner and Perella, 2004). Thus, the analy- sis of the evolution of opinions and concerns about a potential merger is a process similar to rumor verification (Zubiaga et al., 2018b). Notably, the news articles in STANDER discuss the same targets as in WT–WT (Conforti et al., 2020), a large Twitter SD dataset: …

Content truncated to 500 words