SICK-NL: A Dataset for Dutch Natural Language Inference

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1474–1479 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1474 SICK-NL: A Dataset for Dutch Natural Language Inference Gijs Wijnholds UiL-OTS Utrecht University g.j.wijnholds@uu.nl Michael Moortgat UiL-OTS Utrecht University m.j.moortgat@uu.nl Abstract We present SICK-NL (read: signal), dataset targeting Natural Language Inferencea in Dutch. SICK-NL is obtained by translating the SICK dataset of Marelli et al. (2014) from English into Dutch. Having a parallel infer- ence dataset allows us to compare both mono- lingual and multilingual NLP models for En- glish and Dutch on the two tasks. In the pa- per, we motivate and detail the translation pro- cess, perform a baseline evaluation on both the original SICK dataset and its Dutch incarna- tion SICK-NL, taking inspiration from Dutch skipgram embeddings and contextualised em- bedding models. In addition, we encapsulate two phenomena encountered in the translation to formulate stress tests and verify how well the Dutch models capture syntactic restructur- ings that do not affect semantics. Our main finding is all models perform worse on SICK- NL than on SICK, indicating that the Dutch dataset is more challenging than the English original. Results on the stress tests show that models don’t fully capture word order freedom in Dutch, warranting future systematic studies. 1 Introduction One of the primary tasks for Natural Language Processing (NLP) systems is Natural Language In- ference (NLI), where the goal is to determine, for a given premise sentence whether it contradicts, en- tails, or is neutral with respect to a given hypothesis sentence. For English, several standard NLI datasets exist, such as SICK (Marelli et al., 2014), SNLI (Bow- man et al., 2015) and MNLI (Williams et al., 2018). Having such inference datasets available only for English may introduces a bias in NLP research. Conneau et al. (2018) introduce XNLI, a multilin- gual version of a fragment of the SNLI dataset, that contains pairs for Natural Language Inference in 15 languages and is explicitly intended to serve as a re- source for evaluating crosslingual representations. However, Dutch is not represented in any current NLI dataset, a lack that we wish to complement. Dutch counts as a high-resource language, with the sixth largest Wikipedia (2M+ articles), despite having ca. 25M native speakers. Moreover, the syn- tactically parsed LASSY corpus of written Dutch (van Noord et al., 2013), and the SONAR corpus of written Dutch (Oostdijk et al., 2013) provide rich resources on which NLP systems may be devel- oped. Indeed, Dutch is in the scope of the multilin- gual BERT models published by Google (Devlin et al., 2019), and two monolingual Dutch BERT models have been published as part of Hugging- Face’s transformers library (de Vries et al., 2019; Delobelle et al., 2020). Compared to English, however, the number of evaluation tasks for Dutch is limited. There is a Named Entity Recognition task coming from the CoNLL-2003 shared task (Tjong Kim Sang and De Meulder, 2003); from a one million …

Content truncated to 500 words