The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6098–6111, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6098 The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English Francisco Guzm´an♥♦ Peng-Jen Chen♥⋆ Myle Ott⋆ Juan Pino♦ Guillaume Lample⋆‡ Philipp Koehn■ Vishrav Chaudhary♦ Marc’Aurelio Ranzato⋆ ♦Facebook Applied Machine Learning ⋆Facebook AI Research ‡Sorbonne Universit´es ■Johns Hopkins University {fguzman,pipibjc,myleott,juancarabina,guismay,vishrav,ranzato}@fb.com phi@jhu.edu Abstract For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little paral- lel data available. Besides the technical chal- lenges of learning with limited supervision, it is difficult to evaluate methods trained on low- resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLORES evalua- tion datasets for Nepali–English and Sinhala– English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain paral- lel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to col- lect and cross-check the quality of translations, and we report baseline performance using sev- eral learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsu- pervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low- resource MT. Data and code to reproduce our experiments are available at https://github. com/facebookresearch/flores. 1 Introduction Research in Machine Translation (MT) has seen significant advances in recent years thanks to im- provements in modeling, and in particular neural models (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2016; Vaswani et al., 2017), as well as the availability of large parallel cor- pora for training (Tiedemann, 2012; Smith et al., ♥Equal contribution. 2013; Bojar et al., 2017). Indeed, modern neural MT systems can achieve near human-level trans- lation performance on language pairs for which sufficient parallel training resources exist (e.g., Chinese–English translation (Hassan et al., 2018) and English–French translation (Gehring et al., 2016; Ott et al., 2018a). Unfortunately, MT systems, and in particular neural models, perform poorly on low-resource language pairs, for which parallel training data is scarce (Koehn and Knowles, 2017). Improv- ing translation performance on low-resource lan- guage pairs could be very impactful considering that these languages are spoken by a large fraction of the world population. Technically, there are several challenges to solve in order to improve translation for low- resource languages. First, in face of the scarcity of clean parallel data, MT systems should be able to use any source of data available, namely mono- lingual resources, noisy comparable data, as well as parallel data in related languages. Second, we need reliable public evaluation benchmarks to track progress in translation quality. Building evaluation sets on low-resource lan- guages is both expensive and time-consuming be- cause the pool of professional translators is lim- …

Content truncated to 500 words