Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 883–894 November 20–23, 2022. ©2022 Association for Computational Linguistics 883 Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation Hung Thinh Truong1,∗ Yulia Otmakhova1,∗ Timothy Baldwin1,3 Trevor Cohn 1 Jey Han Lau1 Karin Verspoor2,1 1The University of Melbourne 2RMIT University 3MBZUAI {hungthinht,yotmakhova}@student.unimelb.edu.au tb@ldwin.net trevor.cohn@unimelb.edu.au jeyhan.lau@gmail.com karin.verspoor@rmit.edu.au Abstract Negation is poorly captured by current lan- guage models, although the extent of this prob- lem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP meth- ods, with the aim of understanding sub-clausal negation. The test suite contains premise– hypothesis pairs where the premise contains sub-clausal negation and the hypothesis is con- structed by making minimal modifications to the premise in order to reflect different possi- ble interpretations. Aside from adopting stan- dard NLI labels, our test suite is systematically constructed under a rigorous linguistic frame- work. It includes annotation of negation types and constructions grounded in linguistic the- ory, as well as the operations used to construct hypotheses. This facilitates fine-grained analy- sis of model performance. We conduct exper- iments using pre-trained language models to demonstrate that our test suite is more challeng- ing than existing benchmarks focused on nega- tion, and show how our annotation supports a deeper understanding of the current NLI capa- bilities in terms of negation and quantification. 1 Introduction Negation is an important linguistic phenomenon which denotes non-existence, denial, or contradic- tion, and is core to language understanding. NLP work on negation has mostly focused on detecting instances of negation (Peng et al., 2018; Khandel- wal and Sawant, 2020; Truong et al., 2022), and the effect of negation on downstream or probing tasks (Kassner and Schütze, 2020; Ettinger, 2020; Hossain et al., 2020). A consistent finding in recent work on pre-trained language models (PLMs) is that they struggle to correctly handle negation, but also that existing NLP benchmarks are deficient in terms of their relative occurrence and variability *Equal contribution of negation (Barnes et al., 2021; Tang et al., 2021; Hossain et al., 2022). In this work, we address the problem of evaluat- ing the ability of models to handle negation in the English language using a systematic, linguistically- based approach. Specifically, we adopt the typol- ogy proposed by Pullum and Huddleston (2002) whereby negation is classified based on both form (verbal and non-verbal; analytic and synthetic) and meaning (clausal and sub-clausal; ordinary and meta-linguistic). Based on this typology, we ob- serve that most negation instances occurring in ex- isting benchmarks are analytic, verbal, and clausal, which is arguably more straightforward to handle than non-verbal, synthetic, and sub-clausal nega- tion. For instance, the dataset proposed by Hossain et al. (2020) is constructed by adding the syntactic negation cue not to the main verb of the premise and/or the hypothesis of MNLI (Williams et al., …

Content truncated to 500 words