Will-They-Won’t-They: A Very Large Dataset for Stance Detection on Twitter

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1715–1724 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1715 Will-They-Won’t-They: A Very Large Dataset for Stance Detection on Twitter Costanza Conforti1, Jakob Berndt2, Mohammad Taher Pilehvar1,3, Chryssi Giannitsarou2, Flavio Toxvaerd2, Nigel Collier1 1 Language Technology Lab, University of Cambridge 2 Faculty of Economics, University of Cambridge 3 Tehran Institute for Advanced Studies, Iran {cc918,jb2088,mp792,cg349,fmot2,nhc30}@cam.ac.uk Abstract We present a new challenging stance detection dataset, called Will-They-Won’t-They1 (WT– WT), which contains 51,284 tweets in English, making it by far the largest available dataset of the type. All the annotations are carried out by experts; therefore, the dataset constitutes a high-quality and reliable benchmark for future research in stance detection. Our experiments with a wide range of recent state-of-the-art stance detection systems show that the dataset poses a strong challenge to existing models in this domain. The entire dataset is released for future research2. 1 Introduction Apart from constituting an interesting task on its own, stance detection has been identified as a cru- cial sub-step towards many other NLP tasks (Mo- hammad et al., 2017). In fact, stance detection is the core component of fake news detection (Pomer- leau and Rao, 2017), fact-checking (Vlachos and Riedel, 2014; Baly et al., 2018), and rumor verifi- cation (Zubiaga et al., 2018b). Despite its importance, stance detection suffers from the lack of a large dataset which would allow for reliable comparison between models. We aim at filling this gap by presenting Will-They-Won’t- They (WT–WT), a large dataset of English tweets targeted at stance detection for the rumor verifi- cation task. We constructed the dataset based on tweets, since Twitter is a highly relevant platform for rumour verification, which is popular with the public as well as politicians and enterprises (Gor- rell et al., 2019). To make the dataset representative of a realis- tic scenario, we opted for a real-world application 1https://en.wiktionary.org/wiki/will-they-won%27t-they 2https://github.com/cambridge-wtwt/ acl2020-wtwt-tweets of the rumor verification task in finance. Specifi- cally, we constructed the dataset based on tweets that discuss mergers and acquisition (M&A) op- erations between companies. M&A is a general term that refers to various types of financial trans- actions in which the ownership of companies are transferred. An M&A process has many stages that range from informal talks to the closing of the deal. The discussions between companies are usually not publicly disclosed during the early stages of the process (Bruner and Perella, 2004; Piesse et al., 2013). In this sense, the analysis of the evolution of opinions and concerns expressed by users about a possible M&A deal, from its early stage to its closing (or its rejection) stage, is a process similar to rumor verification (Zubiaga et al., 2018a). Moreover, despite the wide interest, most re- search in the intersection of NLP and finance has so far focused on sentiment analysis, text mining and thesauri/taxonomy generation (Fisher et al., 2016; Hahn et al., 2018; El-Haj et al., 2018). While sentiment (Chan and Chong, 2017) and …

Content truncated to 500 words