Understanding by Understanding Not: Modeling Negation in Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1301–1312 June 6–11, 2021. ©2021 Association for Computational Linguistics 1301 Understanding by Understanding Not: Modeling Negation in Language Models Arian Hosseini Mila/Université de Montréal Montréal, Canada arian.hosseini9@gmail.com Siva Reddy Mila/McGill University Montréal, Canada Dzmitry Bahdanau Element AI a ServiceNow Company Montréal, Canada R Devon Hjelm Mila/Université de Montréal and Microsoft Research Montréal, Canada Alessandro Sordoni Microsoft Research Montréal, Canada Aaron Courville Mila/Université de Montréal Montréal, Canada Abstract Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained lan- guage models often handle negation incor- rectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objec- tive that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improve- ments on the negated NLI benchmarks. 1 Introduction Negation is an important property in many lan- guage understanding tasks, such as sentiment anal- ysis, question answering, knowledge base com- pletion and natural language inference (Kassner and Schütze, 2019; Naik et al., 2018). While Pre- trained Language Models (PLMs) such as BERT pushed the state-of-the-art on these tasks (Devlin et al., 2019; Petroni et al., 2019), they fail dra- matically on instances that require understanding negation. Kassner and Schütze (2019) show that current PLMs cannot correctly distinguish between the negated and non-negated forms of fill-in-the-blank tests. For instance, when asked to predict the [MASK] token in sentences such as “The capi- tal of Cuba is [MASK]” and “The capital of Cuba is not [MASK]”, BERT often generate the same answer “Havana”, indicating that it may not be appropriately modeling the distribution of negative sentences. Additional evidence is given by the fact that, when fine-tuned on natural language inference tasks, PLMs tend to mis-classify examples which Figure 1: An overview of the unlikelihood objective. A generic sentence is negated using our data aug- mentation method and an unlikelihood token is cho- sen and replaced with [MASK]. This new sentence is concatenated with the original sentence and fed into the model. The unlikelihood loss is computed using p(improvements) from the language modeling head of BERT. contain not or no as contradiction when the true label is neutral or entailment (Naik et al., 2018). Re- cently, Hossain et al. (2020b) proposed new natural language inference test sets to specifically target the model’s understanding of negation and show that current state-of-the-art models perform poorly on these test sets. In this work, we investigate whether we can alleviate the modeling bias of PLMs on negated sentences. Our approach is composed of two core contributions: i) a syntactic data augmenta- tion scheme to automatically generate negated sen- tences; ii) a new training paradigm, dubbed unlike- lihood training with reference (Fig. 1), based on the recently …

Content truncated to 500 words