Context Matters: A Pragmatic Study of PLMs’ Negation Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4602 - 4621 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Context Matters: A Pragmatic Study of PLMs’ Negation Understanding Reto Gubelmann University of St.Gallen Rosenbergstrasse 30 9012 St.Gallen reto.gubelmann@unisg.ch Siegfried Handschuh University of St.Gallen Rosenbergstrasse 30 9012 St.Gallen siegfried.handschuh@unisg.ch Abstract In linguistics, there are two main perspectives on negation: a semantic and a pragmatic view. So far, research in NLP on negation has almost exclusively adhered to the semantic view. In this article, we adopt the pragmatic paradigm to conduct a study of negation understanding fo- cusing on transformer-based PLMs. Our results differ from previous, semantics-based studies and therefore help to contribute a more com- prehensive – and, given the results, much more optimistic – picture of the PLMs’ negation un- derstanding. 1 Introduction Transformer-Based pre-trained language models (PLMs) have become the de facto standard in a variety of natural language processing tasks. Based on the original transformer architecture (Vaswani et al., 2017), researchers have proposed a number of extraordinarily successful architectures, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and smaller ver- sions such as DistilBERT (Sanh et al., 2019). Such transformer-based models perform impressively at standard natural language understanding (NLU) benchmarks, often outperforming the human bench- mark, as evinced by the GLUE and SuperGLUE Leaderboards (Wang et al., 2018 and Wang et al., 2019). While it is impossible to deny the performance of these models at such benchmarks, it is another, particularly challenging question whether this per- formance is driven by simple shallow heuristics or by any real understanding of the languages that they are processing.1 This study contributes to an- swering this question with a focus on negation. Answering the question is important for both theoretical and practical reasons. On the theoreti- 1Compare the appendix, section A, for a more elaborate sketch of the theoretical background of our concept of real understanding. cal side, the study contributes to a more accurate understanding of the driving forces behind the pre- dictions issued by PLMs. The phenomenon of negation, being both highly semantically relevant and having a small footprint on the syntactic sur- face of a sentence, is ideally suited for this purpose. For real-world applications, it is crucial to know whether the model predicts based on simple con- textual clues or on a real understanding of nega- tion. For instance, for NLU applications, it makes a substantial difference whether a certain microblog recommends that one should “(not) get vaccinated against covid-19”. The question is challenging for two main rea- sons. First, because the models’ performance is typically strong. However, it is wrong, not cor- rect predictions that potentially unveil underlying heuristics. Second, as we will show, it requires careful, linguistically conscientious construction of the datasets to be able to draw sound conclusions even from wrong predictions. For example, if a model predicts “fly” in example (1), this has been taken by …

Content truncated to 500 words