CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8729 - 8755 December 7-11, 2022 ©2022 Association for Computational Linguistics CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation Abhilasha Ravichander∗ Carnegie Mellon University aravicha@cs.cmu.edu Matt Gardner Microsoft Semantic Machines mattgardner@microsoft.com Ana Marasovi´c∗ University of Utah ana.marasovic@utah.edu Abstract The full power of human language-based com- munication cannot be realized without negation. All human languages have some form of nega- tion. Despite this, negation remains a challeng- ing phenomenon for current natural language understanding systems. To facilitate the fu- ture development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implica- tions of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage—paraphrasing the negated statement, changing the scope of the negation, and reversing the negation—resulting in clusters of question-answer pairs that are dif- ficult for models to answer with spurious short- cuts. CONDAQA features 14,182 question- answer pairs with over 200 unique negation cues and is challenging for current state-of- the-art models. The best performing model on CONDAQA (UNIFIEDQA-V2-3B) achieves only 42% on our consistency metric, well be- low human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facili- tate the development of future NLP methods that work on negated language. 1 Introduction Negation is fundamental to human communication. It is a phenomenon of semantic opposition, relat- ing one expression to another whose meaning is in some way opposed. Negation supports key proper- ties of human linguistic systems such as contradic- tion and denial (Horn, 1989). *Work undertaken while Abhilasha Ravichander and Ana Marasovi´c were at the Allen Institute for AI. Despite the prevalence of negation, processing it effectively continues to elude models. Here are just a few of the many recently reported failures: “The model [BERT-Large trained on SQuAD] does not seem capable of handling...simple examples of negation” (Ribeiro et al., 2020). “We find that indeed the presence of negation can significantly impact downstream quality [of machine translation systems]” (Hossain et al., 2020a). “State-of-the-art models answer questions from the VQA...correctly, but struggle when asked a logical composition in- cluding negation” (Gokhale et al., 2020). How can NLU systems meet this long-standing challenge? To facilitate systems that can process negation effectively, it is crucial to have high-quality eval- uations that accurately measure models’ compe- tency at processing and understanding negation. In this work, we take a step toward this goal by contributing the first large-scale reading compre- hension dataset, CONDAQA, focused on reasoning about negated statements in language.1 The three-stage annotation process we develop to construct CONDAQA is illustrated in Fig. 1. We first collect passages from English Wikipedia that contain negation cues, including single- and multi- word negation phrases, as well as affixal negation. In the …

Content truncated to 500 words