ConjNLI: Natural Language Inference Over Conjunctive Sentences

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8240–8252, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8240 CONJNLI: Natural Language Inference Over Conjunctive Sentences Swarnadeep Saha Yixin Nie Mohit Bansal UNC Chapel Hill {swarna, yixin1, mbansal}@cs.unc.edu Abstract Reasoning about conjuncts in conjunctive sen- tences is important for a deeper understanding of conjunctions in English and also how their usages and semantics differ from conjunctive and disjunctive boolean logic. Existing NLI stress tests do not consider non-boolean us- ages of conjunctions and use templates for testing such model knowledge. Hence, we introduce CONJNLI, a challenge stress-test for natural language inference over conjunc- tive sentences, where the premise differs from the hypothesis by conjuncts removed, added, or replaced. These sentences contain single and multiple instances of coordinating con- junctions (“and”, “or”, “but”, “nor”) with quantifiers, negations, and requiring diverse boolean and non-boolean inferences over con- juncts. We find that large-scale pre-trained language models like RoBERTa do not under- stand conjunctive semantics well and resort to shallow heuristics to make inferences over such sentences. As some initial solutions, we first present an iterative adversarial fine-tuning method that uses synthetically created training data based on boolean and non-boolean heuris- tics. We also propose a direct model advance- ment by making RoBERTa aware of predicate semantic roles. While we observe some perfor- mance gains, CONJNLI is still challenging for current methods, thus encouraging interesting future work for better understanding of con- junctions.1 1 Introduction Coordinating conjunctions are a common syntac- tic phenomenon in English: 38.8% of sentences in the Penn Tree Bank have at least one coordinating word between “and”, “or”, and “but” (Marcus et al., 1993). Conjunctions add complexity to the sen- tences, thereby making inferences over such sen- 1CONJNLI data and code are publicly available at https: //github.com/swarnaHub/ConjNLI. tences more realistic and challenging. A sentence can have many conjunctions, each conjoining two or more conjuncts of varied syntactic categories such as noun phrases, verb phrases, prepositional phrases, clauses, etc. Besides syntax, conjunctions in English have a lot of semantics associated to them and different conjunctions (“and” vs “or”) affect the meaning of a sentence differently. Recent years have seen significant progress in the task of Natural Language Inference (NLI) through the development of large-scale datasets like SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). Although large-scale pre-trained language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have achieved super-human performances on these datasets, there have been concerns raised about these models exploiting idiosyncrasies in the data using tricks like pattern matching (McCoy et al., 2019). Thus, various stress-testing datasets have been proposed that probe NLI models for simple lexical inferences (Glockner et al., 2018), quan- tifiers (Geiger et al., 2018), numerical reasoning, antonymy and negation (Naik et al., 2018). How- ever, despite the heavy usage of conjunctions in English, there is no specific NLI dataset that tests their understanding in detail. Although SNLI has 30% of samples with conjunctions, …

Content truncated to 500 words