Assessing Monotonicity Reasoning in Dutch through Natural Language Inference

Findings of the Association for Computational Linguistics: EACL 2023, pages 1494–1500 May 2-6, 2023 ©2023 Association for Computational Linguistics Assessing Monotonicity Reasoning in Dutch through Natural Language Inference Gijs Wijnholds Institute for Language Sciences Utrecht University g.j.wijnholds@uu.nl Abstract In this paper we investigate monotonicity rea- soning in Dutch, through a novel Natural Lan- guage Inference dataset. Monotonicity rea- soning shows to be highly challenging for Transformer-based language models in English and here, we corroborate those findings using a parallel Dutch dataset, obtained by translating the Monotonicity Entailment Dataset of Yanaka et al. (2019). After fine-tuning two Dutch lan- guage models BERTje and RobBERT on the Dutch NLI dataset SICK-NL, we find that per- formance severely drops on the monotonicity reasoning dataset, indicating poor generaliza- tion capacity of the models. We provide a de- tailed analysis of the test results by means of the linguistic annotations in the dataset. We find that models struggle with downward en- tailing contexts, and argue that this is due to a poor understanding of negation. Additionally, we find that the choice of monotonicity context affects model performance on conjunction and disjunction. We hope that this new resource paves the way for further research in general- ization of neural reasoning models in Dutch, and contributes to the development of better language technology for Natural Language In- ference, specifically for Dutch. 1 Introduction Natural Language Inference (NLI) is one of the standard benchmark tasks for current-day NLP ar- chitectures. In this task a model takes two sen- tences as input, and has to classify the relationship between the former (premise) sentence and the lat- ter (hypothesis) sentence, typically between Entail- ment, Contradiction, and Neutral. NLI makes for a interesting task as drawing the correct inference may require subtle aspects of syntax, lexical se- mantics, and even pragmatics. While many NLI datasets exist like SICK (Marelli et al., 2014), SNLI (Bowman et al., 2015) and its extensions (MNLI Williams et al. (2018), XNLI Conneau et al. (2018), e-SNLI Camburu et al. (2018)), much is still un- known about how and why neural language models (LMs) like BERT (Devlin et al., 2019) perform on the task. Evidence shows that fine-tuned LMs don’t generalize well across NLI benchmarks (Talman and Chatzikyriakidis, 2019), and other investiga- tion shows that LMs may be exploiting dataset heuristics to solve the task (Naik et al., 2018; Mc- Coy et al., 2019). More generally, LMs do seem to encode a certain amount of syntactic structure (Rogers et al., 2020), but the relation to NLI re- mains unclear. In order to shed light on the performance of large- scale LMs, specific datasets have been developed to understand what models do and don’t under- stand. Specifically in the context of NLI, Yanaka et al. (2019) introduce the Monotonicity Entailment Dataset (MED), which targets models’ capacity for understanding monotonicity reasoning (Icard III and Moss, 2014). Monotonicity reasoning is a sta- ple test of human reasoning which requires lexical knowledge, as well as syntactic knowledge, making it suitable for …

Content truncated to 500 words