The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1174–1182 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1174 The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT J¨org Tiedemann University of Helsinki jorg.tiedemann@helsinki.fi https://github.com/Helsinki-NLP/Tatoeba-Challenge Abstract This paper describes the development of a new benchmark for machine translation that pro- vides training and test data for thousands of language pairs covering over 500 languages and tools for creating state-of-the-art transla- tion models from that collection. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World’s languages. Using the package it is possible to work on realistic low-resource scenarios avoiding arti- ficially reduced setups that are common when demonstrating zero-shot or few-shot learning. For the first time, this package provides a comprehensive collection of diverse data sets in hundreds of languages with systematic lan- guage and script annotation and data splits to extend the narrow coverage of existing bench- marks. Together with the data release, we also provide a growing number of pre-trained base- line models for individual language pairs and selected language groups. 1 Introduction The Tatoeba translation challenge includes shuffled training data taken from OPUS,1 an open collec- tion of parallel corpora (Tiedemann, 2012), and test data from Tatoeba,2 a crowd-sourced collec- tion of user-provided translations in a large number of languages. All data sets are labeled with ISO- 639-3 language codes using macro-languages in case when available. Naturally, training data do not include sentences from Tatoeba and neither from the popular WMT testsets to allow a fair compari- son to other models that have been evaluated using those data sets. Here, we propose an open challenge and the idea is to encourage people to develop machine transla- tion in real-world cases for many languages. The 1http://opus.nlpl.eu/ 2https://tatoeba.org/ most important point is to get away from artificial setups that only simulate low-resource scenarios or zero-shot translations. A lot of research is tested with multi-parallel data sets and high resource lan- guages using data sets such as WIT3 (Cettolo et al., 2012) or Europarl (Koehn, 2005) simply reducing or taking away one language pair for arguing about the capabilities of learning translation with little or without explicit training data for the language pair in question (see, e.g., Firat et al. (2016a,b); Ha et al. (2016); Lakew et al. (2018)). Such a setup is, how- ever, not realistic and most probably over-estimates the ability of transfer learning making claims that do not necessarily carry over towards real-world tasks. In the set we provide here we, instead, include all available data from the collection without re- moving anything. In this way, the data refers to a diverse and skewed collection, which reflects the real situation we need to work with and many low- resource languages are only represented by noisy or very unrelated training data. Zero-shot scenar- ios are only tested if no data is available in any of the sub-corpora. More details …

Content truncated to 500 words