Spurious Correlations in Cross-Topic Argument Mining

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 263–277 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 263 Spurious Correlations in Cross-Topic Argument Mining Terne Sasha Thorn Jakobsen1 Maria Barrett2 Anders Søgaard 3 1Copenhagen Center for Social Data Science, University of Copenhagen terne.thorn@sodas.ku.dk 2Computer Science Department, IT University of Copenhagen mbarrett@itu.dk 3Department of Computer Science, University of Copenhagen soegaard@di.ku.dk Abstract Recent work in cross-topic argument mining attempts to learn models that generalise across topics rather than merely relying on within- topic spurious correlations. We examine the effectiveness of this approach by analysing the output of single-task and multi-task models for cross-topic argument mining through a com- bination of linear approximations of their de- cision boundaries, manual feature grouping, challenge examples, and ablations across the input vocabulary. Surprisingly, we show that cross-topic models still rely mostly on spu- rious correlations and only generalise within closely related topics, e.g., a model trained only on closed-class words and a few common open-class words outperforms a state-of-the- art cross-topic model on distant target topics. 1 Introduction When a sentiment analysis model associates the word Shrek with positive sentiment (Sindhwani and Melville, 2008), it relies on a spurious cor- relation. While the movie Shrek was popular at the time the training data was sampled, this is un- likely to transfer across demographics, platforms and years. While there exists a continuum from sentiment words such as fantastic to spurious cor- relations such as Shrek, with words such as Holly- wood or anticipation being perhaps in a grey zone, demoting spurious correlations is key to learning robust NLP models (Sutton et al., 2006; Søgaard, 2013; Tu et al., 2020). This paper studies a similar problem in state-of- the-art cross-topic argument mining systems. The task of argument mining is to recognise the exis- tence of claims and premises in a text span. The All code will be publicly available at https:// github.com/terne/spurious_correlations_ in_argmin Figure 1: In human interaction, it is evident that rely- ing on topic words for recognizing an argument is non- sensical. It is, nevertheless, what a BERT-based cross- topic argument mining model does. standard evaluation protocol is to evaluate argu- ment mining systems across topics, i.e., on held- out topics, precisely to avoid over-fitting to a single topic (Daxenberger et al., 2017; Stab et al., 2018; Reimers et al., 2019). This study shows that despite this sensible cross-topic evaluation protocol, state- of-the-art systems nevertheless rely primarily on spurious correlations, e.g., guns (Figure 1). These spurious correlations transfer across some topics in popular benchmarks, but only because the topics are closely related. Contributions We present experiments with an out-of-the-box learning architecture for argument mining, yet with state-of-the-art performance, based on Microsoft’s MT-DNN library (Liu et al., 2019). We train models on the UKP Sentential Argument Mining Corpus (Stab et al., 2018), the IBM Debater Argument Search Engine Dataset (Levy et al., 2018), the Argument Extraction corpus (Swanson et al., 2015), and the Vaccination Corpus (Morante et al., 2020). We analyse …

Content truncated to 500 words