Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13448–13464 July 9-14, 2023 ©2023 Association for Computational Linguistics Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection Erik Arakelyan1, Arnav Arora2, Isabelle Augenstein3 Department of Computer Science University of Copenhagen Copenhagen Denmark {erik.a,aar,augenstein}@di.ku.dk Abstract Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data col- lection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly im- balanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose Topic Efficient StancE Detection (TESTED), consisting of a topic-guided diver- sity sampling technique and a contrastive ob- jective that is used for fine-tuning a stance clas- sifier. We evaluate the method on an existing benchmark of 16 datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of 3.5 F1 points increase in-domain, and is more generalizable with an averaged increase of 10.2 F1 on out-of-domain evalua- tion while using ≤ 10% of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Fi- nally, our analysis demonstrates that the con- trastive learning objective allows the model a more pronounced segmentation of samples with varying labels. 1 Introduction The goal of stance detection is to identify the viewpoint expressed by an author within a piece of text towards a designated topic (Mohammad et al., 2016). Such analyses can be used in a va- riety of domains ranging from identifying claims within political or ideological debates (Somasun- daran and Wiebe, 2010; Thomas et al., 2006), iden- tifying mis- and disinformation (Hanselowski et al., 2018; Hardalov et al., 2022a), public health pol- icymaking (Glandt et al., 2021; Hossain et al., 2020; Osnabrügge et al., 2023), news recommenda- tion (Reuver et al., 2021) to investigating attitudes voiced on social media (Qazvinian et al., 2011; Au- genstein et al., 2016; Conforti et al., 2020). How- ever, in most domains, and even more so for cross- domain stance detection, the exact formalisation of the task gets blurry, with varying label sets and their corresponding definitions, data collection pro- tocols and available annotations. Furthermore, this is accompanied by significant changes in the topic- specific vocabulary (Somasundaran and Wiebe, 2010; Wei and Mao, 2019), text style (Pomerleau and Rao, 2017; Ferreira and Vlachos, 2016) and topics mentioned either explicitly (Qazvinian et al., 2011; Walker et al., 2012) or implicitly (Hasan and Ng, 2013; Derczynski et al., 2017). Recently, a benchmark of 16 datasets (Hardalov et al., 2021) covering a variety of domains and topics has been proposed for …

Content truncated to 500 words