InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples

Proceedings of the 29th International Conference on Computational Linguistics, pages 3873–3884 October 12–17, 2022. 3873 INFERES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples Venelin Kovatchev School of Information The University of Texas at Austin venelin@utexas.edu Mariona Taulé Centre de Llenguatge i Computació Institut de Recerca en Sistemes Complexos Universitat de Barcelona mtaule@ub.edu Abstract In this paper, we present INFERES - an orig- inal corpus for Natural Language Inference (NLI) in European Spanish. We propose, imple- ment, and analyze a variety of corpus-creating strategies utilizing expert linguists and crowd workers. The objectives behind INFERES are to provide high-quality data, and, at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to gener- alize across out-of-distribution topics. We train two transformer models on IN- FERES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% ac- curacy, leaving a lot of room for improvement. The “hypothesis-only” baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We find that models trained on INFERES general- ize very well across topics (both in- and out-of- distribution) and perform moderately well on negation-based adversarial examples. 1 Introduction In the task of Natural Language Inference (NLI), an automated system has to determine the meaning relation that holds between two texts. The model has to make a three-way choice between entailment: a hypothesis (h) is true given a premise (p) (e.g. 1.); contradiction: a hypothesis (h) is false given a premise (p) (e.g. 2.); or neutral: the truth value of the hypothesis (h) cannot be determined solely based on the premise (p) (e.g.: 3.). 1. p) John goes to work every day with a car. h) John has a job. 2. p) John goes to work every day with a car. h) John takes the bus to go to work. 3. p) John goes to work every day with a car. h) John has a Porsche. NLI (formerly known as Recognizing Textual Entailment (RTE)) is one of the core tasks in the popular benchmarks for Natural Language Under- standing GLUE (Wang et al., 2018) and Super GLUE (Wang et al., 2019). Hundreds of machine learning systems compete on these benchmarks, improving the state of NLU. One key limitation of NLI research is that most of the existing corpora are only for English. Lim- ited research has been done on multilingual and non-English corpora (Peñas et al., 2006; Conneau et al., 2018; Amirkhani et al., 2020; Ham et al., 2020; Hu et al., 2020; Mahendra et al., 2021). Another well-known issue with NLI is the qual- ity of the existing datasets and the limitations of the models trained on them. On most NLI cor- pora, state-of-the-art transformer based models can obtain quantitative results (Accuracy and F1) that equal or exceed human performance. Despite this high performance, researchers have identified nu- merous …

Content truncated to 500 words