Data and Representation for Turkish Natural Language Inference

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8253–8267, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8253 Data and Representation for Turkish Natural Language Inference Emrah Budur,1,2 Rıza ¨Ozc¸elik,2 Tunga G¨ung¨or,2 and Christopher Potts3 1Garanti BBVA Technology 2Bo˘gazic¸i University 3Stanford University {emrah.budur, riza.ozcelik, gungort}@boun.edu.tr, cgpotts@stanford.edu Abstract Large annotated datasets in NLP are over- whelmingly in English. This is an obstacle to progress in other languages. Unfortunately, ob- taining new annotated resources for each task in each language would be prohibitively ex- pensive. At the same time, commercial ma- chine translation systems are now robust. Can we leverage these systems to translate English- language datasets automatically? In this paper, we offer a positive response for natural lan- guage inference (NLI) in Turkish. We trans- lated two large English NLI datasets into Turk- ish and had a team of experts validate their translation quality and fidelity to the origi- nal labels. Using these datasets, we address core issues of representation for Turkish NLI. We find that in-language embeddings are es- sential and that morphological parsing can be avoided where the training set is large. Finally, we show that models trained on our machine- translated datasets are successful on human- translated evaluation sets. We share all code, models, and data publicly. 1 Introduction Many tasks in natural language processing have been transformed by the introduction of very large annotated datasets. Prominent examples include paraphrase (Ganitkevitch et al., 2013), parsing (Nivre et al., 2016), question answering (Rajpurkar et al., 2016), machine translation (MT; Bojar et al., 2014), and natural language inference (NLI; Bow- man et al., 2015; Williams et al., 2018a). Unfortunately, outside of parsing and MT, these datasets tend to be in English. This is not only an obstacle to progress on other languages, but it also limits the field of NLP itself: English is gen- erally not a representative example of the world’s languages when it comes to morphology, syntax, or spelling conventions and other kinds of standard- ization (Munro, 2012), so it’s risky to assume that models and results for English will generalize to other languages. A natural response to these gaps in our dataset coverage might be to launch new annotation efforts for multiple languages. However, this would likely be prohibitively expensive. For example, based on the costs of SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018a), we estimate that each large dataset for NLI would cost upwards of US $50,000 if created completely from scratch. At the same time, commercial MT systems have improved dramatically in recent years (Wu et al., 2016; Johnson et al., 2017; Hieber et al., 2017, 2018; Tomasello, 2019; Hieber et al., 2020). They now offer high-quality translations between hun- dreds of language pairs. This raises the question: can we use these MT systems to translate English- language datasets and use the translated versions to drive more genuinely multilingual development in NLP? In this paper, we offer evidence that the answer is “yes”. Using Amazon Translate, …

Content truncated to 500 words