tWT–WT: A Dataset to Assert the Role of Target Entities for Detecting Stance of Tweets

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3879–3889 June 6–11, 2021. ©2021 Association for Computational Linguistics 3879 tWT–WT: A Dataset to Assert the Role of Target Entities for Detecting Stance of Tweets Ayush Kaushal and Avirup Saha and Niloy Ganguly Indian Institute of Technology Kharagpur {ayushk4, saha.avirup}@gmail.com, niloy@cse.iitkgp.ac.in Abstract The stance detection task aims at detecting the stance of a tweet or a text for a target. These targets can be named entities or free-form sen- tences (claims). Though the task involves rea- soning of the tweet with respect to a target, we find that it is possible to achieve high accuracy on several publicly available Twitter stance de- tection datasets without looking at the target sentence. Specifically, a simple tweet clas- sification model achieved human-level perfor- mance on the WT–WT dataset and more than two-third accuracy on various other datasets. We investigate the existence of biases in such datasets to find the potential spurious corre- lations of sentiment-stance relations and lexi- cal choice associated with the stance category. Furthermore, we propose a new large dataset free of such biases and demonstrate its apt- ness on the existing stance detection systems. Our empirical findings show much scope for research on the stance detection task and pro- poses several considerations for creating fu- ture stance detection datasets.1 1 Introduction Stance detection is a vital sub-task for fake news detection (Pomerleau and Rao, 2017), automated fact checking (Vlachos and Riedel, 2014; Ferreira and Vlachos, 2016), social media analysis (Zhang et al., 2017), analyzing online debates (Bar-Haim et al., 2017) and rumour verification (Derczynski et al., 2017; Gorrell et al., 2019). Furthermore, it is also an essential measure for progress in Natural Language Understanding, especially in the noisy- text domain. Over the recent years, several stance detection datasets have been proposed. These datasets, in turn, facilitated progress in stance detection re- search, with some systems achieving up to 93.7% accuracy (Dulhanty et al., 2019). However, most 1Code: https://github.com/Ayushk4/bias-stance Dataset: https://github.com/Ayushk4/stance-dataset Analysts: Aetna-Humana Deal still Probable, Anthem-Cigna Unlikely - @InsuranceNewsNet <URL> 11:03 AM • • Twitter for iPhone [Aetna-Humana] Support [Anthem-Cigna] Refute Figure 1: An example tweet from WT–WT dataset with different targets. of these state-of-the-art systems are complex deep neural networks, making them difficult to interpret. Lack of explainability raises concern since previous works (Gururangan et al., 2018; Goyal et al., 2017; Cirik et al., 2018; Geva et al., 2019) on other tasks demonstrated that superficial dataset biases could result in inflated test-set performance. With this motivation, we carry out the first study analyzing several publicly available Twitter stance detection datasets. Our experiments reveal rampant biases in datasets through which even target-oblivious mod- els can achieve impressive performance. Various existing works have hinted at the pres- ence of such dataset biases. For example, TAN model (Du et al., 2017) is a very competitive stance detection model.However, Ghosh et al. (2019) re- cently proved that TAN does not take advantage of …

Content truncated to 500 words