How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 226 - 233 July 14-15, 2022 ©2022 Association for Computational Linguistics How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE Datasets Aarne Talman*§ Marianna Apidianaki⋄ Stergios Chatzikyriakidis†‡§ Jörg Tiedemann* *Department of Digital Humanities, University of Helsinki {name.surname}@helsinki.fi ⋄Department of Computer and Information Science, University of Pennsylvania marapi@seas.upenn.edu †Department of Philology, University of Crete stergios.chatzikyriakidis@uoc.gr ‡Centre of Linguistic Theory and Studies in Probability, FLoV, University of Gothenburg §Basement AI Abstract A central question in natural language under- standing (NLU) research is whether high perfor- mance demonstrates the models’ strong reason- ing capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sen- tences. Our results show that performance re- mains high on most GLUE tasks when the mod- els are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset con- stitutes a proper testbed for evaluating models’ language understanding capabilities. 1 Introduction The super-human performance of recent Transformer-based pre-trained language models (Devlin et al., 2019; Liu et al., 2019) on natural language understanding (NLU) tasks has raised scepticism regarding the quality of the benchmarks used for evaluation (Wang et al., 2018, 2019). There is increasing evidence that these datasets contain annotation artefacts and other statistical irregularities that can be leveraged by machine learning models to perform the tasks (Gururangan et al., 2018; Poliak et al., 2018b; Tsuchiya, 2018; Glockner et al., 2018; Talman and Chatzikyriakidis, 2019; Pham et al., 2020; Talman et al., 2021). These studies have so far largely focused on the natural language inference (NLI) and textual entailment tasks. The scope of our work is wider, in the sense that we address all but one NLU tasks Sentence 1 Sentence 2 paraphrase Easynews Inc. was subpoenaed late last week by the FBI, which was seeking account information related to the uploading of the virus to the ISP’s Usenet news group server. Easynews Inc. said Monday that it was co- operating with the FBI in trying to locate the person who uploaded the virus to a Usenet news group hosted by the ISP. non-paraphrase Arison said Mann may have been one of the pioneers of the world music movement and he had a deep love of Brazilian music. Arison said Mann was a pioneer of the world music movement – well before the term was coined – and he had a deep love of Brazilian music. Table 1: Example sentence pairs from the corrupted MRPC training dataset where all instances of nouns have been removed. comprised in the GLUE benchmark, specifically: linguistic acceptability (COLA), paraphrasing (MRPC and QQP), sentiment prediction (SST-2), and semantic textual similarity (STS-B). We present a series of experiments where …

Content truncated to 500 words