An Analysis of Natural Language Inference Benchmarks through the Lens of Negation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9106–9118, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9106 An Analysis of Natural Language Inference Benchmarks through the Lens of Negation Md Mosharaf Hossain,8 Venelin Kovatchev,Z Pranoy Dutta,8 Tiffany Kao,8 Elizabeth Wei,8 and Eduardo Blanco8 8University of North Texas ZUniversity of Barcelona mdmosharafhossain@my.unt.edu vkovatchev@ub.edu {PranoyDutta,TiffanyKao,ElizabethWei}@my.unt.edu eduardo.blanco@unt.edu Abstract Negation is underrepresented in existing natural language inference benchmarks. Additionally, one can often ignore the few negations in existing benchmarks and still make the right inference judgments. In this paper, we present a new benchmark for natural language inference in which negation plays an important role. We also show that state-of-the-art transformers struggle making inference judgments with the new pairs. 1 Introduction Natural language understanding remains an elu- sive goal except in limited scenarios. It is arguably the ultimate problem in natural language process- ing: to empower machines to understand language as generated by humans. The state of the art has seen tremendous progress in recent years, and has moved from symbolic representations (Bos et al., 2004; Artzi and Zettlemoyer, 2013) to distribu- tional representations often learned from massive datasets (Devlin et al., 2019). Recognizing en- tailments (Dagan et al., 2006), identifying para- phrases (Das and Smith, 2009), determining se- mantic textual similarity (Agirre et al., 2012), and sentiment analysis (Pang and Lee, 2008) are but a few problems that require natural language under- standing to a lesser or greater degree. There are many benchmarks targeting the prob- lems above, and they usually cast them as classifi- cation problems. A couple of popular evaluation platforms, GLUE (Wang et al., 2018) and Super- GLUE (Wang et al., 2019), aggregate benchmarks for some of the problems above and provide a sin- gle score for many tasks under the umbrella of natu- ral language inference. State-of-the-art models are close to or even surpass human performance (Wang et al., 2019). This fact, however, is true only when evaluating models and humans with existing bench- marks. Indeed, researchers have pointed out weak- nesses in benchmarks suggesting that we are eval- uating models with examples that are much sim- pler than what humans are capable of (Section 3). Source text selection, annotation artifacts (Guru- rangan et al., 2018), and asking annotators—either experts or crowd workers—to write examples as op- posed to retrieving real examples from previously generated language are a few of the culprits. In this paper, we investigate the role of negation in a core natural language understanding task: nat- ural language inference—in its most basic form, determining whether a text entails a hypothesis. Recognizing entailments has many applications in- cluding question answering (Trivedi et al., 2019), summarization (Pasunuru et al., 2017) and machine translation evaluation (Pad´o et al., 2009). Negation relates an expression e to another ex- pression with a meaning that is in some way op- posed to the meaning of e (Horn and Wansing, 2017), thus it plays an important role in natural language understanding. Additionally, negation is ubiquitous in regular …

Content truncated to 500 words