Baselines and Test Data for Cross-Lingual Inference

Baselines and Test Data for Cross-Lingual Inference ˇZeljko Agi´c Natalie Schluter Department of Computer Science IT University of Copenhagen Rued Langgaards Vej 7, 2300 Copenhagen S, Denmark zeag@itu.dk nael@itu.dk Abstract The recent years have seen a revival of interest in textual entailment, sparked by i) the emergence of powerful deep neural network learners for natural language processing and ii) the timely development of large-scale evaluation datasets such as SNLI. Recast as natural language inference, the problem now amounts to detecting the relation between pairs of statements: they either contradict or entail one another, or they are mutually neutral. Current research in natural language inference is effectively exclusive to English. In this paper, we propose to advance the research in SNLI-style natural language inference toward multilingual evaluation. To that end, we provide test data for four major languages: Arabic, French, Spanish, and Russian. We experiment with a set of baselines. Our systems are based on cross-lingual word embeddings and machine translation. While our best system scores an average accuracy of just over 75%, we focus largely on enabling further research in multilingual inference. Keywords: natural language inference, cross-lingual methods, test data 1. Introduction Natural language processing is marking a very recent resur- gence of interest in textual entailment. Now revamped as natural language inference (NLI) by Bowman et al. (2015) with their SNLI dataset, the task of differentiating contradictory, entailing, and unrelated pairs of sentences (Fig. 1) has entertained a large number of proposals.1 The timely challenge lends itself to various deep learning ap- proaches such as by Rockt¨aschel et al. (2015), Parikh et al. (2016), or Wang et al. (2017), which mark a string of very notable results. Yet, the SNLI corpus is in English only. As of recently, it includes more test data from multiple genres,2 but it re- mains exclusive to English. Following Bender (2009) in seeking true language independence, we propose to extend the current NLI research beyond English, and further into the majority realm of low-resource languages. Since training data is generally unavailable for most lan- guages, work on transfer learning is abundant for the basic NLP tasks such as tagging and syntactic parsing (Das and Petrov, 2011; Ammar et al., 2016). By contrast, the re- search in cross-lingual entailment is not as plentiful (Negri et al., 2013). To the best of our knowledge, at this point there are no contributions to SNLI-style cross-lingual in- ference, or for that matter, work on languages other than English at all. Contributions. In the absence of training data for lan- guages other than English, we propose a set of baselines for cross-lingual neural inference. We adapt to the target languages either by i) employing multilingual word embed- dings or alternatively by ii) translating the input sentebces into English. We create multilingual test data to facilitate evaluation by manually translating 4 × 1,332 premise-hypothesis sen- tence pairs from the English SNLI test data into four other major languages: Arabic, French, Russian, and Spanish. 1https://nlp.stanford.edu/projects/snli/ 2https://repeval2017.github.io/shared/ premise Female gymnasts warm up …

Content truncated to 500 words