On Classifying whether Two Texts are on the Same Side of an Argument

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10130–10138 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10130 On Classifying whether Two Texts are on the Same Side of an Argument Erik Körner † Gregor Wiedemann ‡ Ahmad Dawar Hakimi † Gerhard Heyer † Martin Potthast † †Leipzig University ‡Leibniz-Institute for Media Research | Hans-Bredow-Institut Abstract To ease the difficulty of argument stance clas- sification, the task of same side stance classi- fication (S3C) has been proposed. In contrast to actual stance classification, which requires a substantial amount of domain knowledge to identify whether an argument is in favor or against a certain issue, it is argued that, for S3C, only argument similarity within stances needs to be learned to successfully solve the task. We evaluate several transformer-based approaches on the dataset of the recent S3C shared task, followed by an in-depth evalua- tion and error analysis of our model and the task’s hypothesis. We show that, although we achieve state-of-the-art results, our model fails to generalize both within as well as across topics and domains when adjusting the sam- pling strategy of the training and test set to a more adversarial scenario. Our evaluation shows that current state-of-the-art approaches cannot determine same side stance by consid- ering only domain-independent linguistic sim- ilarity features, but appear to require domain knowledge and semantic inference, too. 1 Introduction Same side stance classification (S3C) is the task to predict, for a given pair of arguments, whether both argue for the same stance (Stein et al., 2021). It abstracts from conventional stance classification, which, for an individual argument, predicts whether it argues for or against a corresponding issue. The hypothesis underlying S3C is that it can “probably be solved independently of a topic or a domain, so to speak, in a topic-agnostic fashion”.1 Successful S3C can, for instance, help to quickly identify co- herent posts in social media debates, or to quantify groups of posts with opposing stances. To advance S3C as a task in the argument mining community, this paper makes three main contributions: (1) De- velopment of new transformer-based approaches 1More details on the task at https://sameside.webis.de which improve upon the state of the art. (2) Re- newed assessment of the original S3C shared task dataset, and compilation of new training and test sets that enable a more realistic evaluation scenario. (3) Compilation of an additional, hand-crafted test set consisting of adversarial cases, such as nega- tions and references to contrary positions within single arguments, to investigate the hypothesis un- derlying S3C in particular. Our results indicate that current state-of-the-art models are not able to solve such cases. We conclude with recommendations how datasets and evaluation scenarios for the S3C task could be further developed.2 2 Related Work Stance Classification S3C has been introduced as a shared task by Stein et al. (2021). Prior work on stance classification, such as that of Somasun- daran and Wiebe (2010), Gottipati et al. (2013), and Sridhar et al. (2015), focuses …

Content truncated to 500 words