T4T Solution: WMT21 Similar Language Task for the Spanish-Catalan and Spanish-Portuguese Language Pair

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 89–99 November 10–11, 2021. ©2021 Association for Computational Linguistics 89 Findings of the WMT 2021 Shared Task on Large-Scale Multilingual Machine Translation Guillaume Wenzek, Vishrav Chaudhary, Angela Fan, Sahir Gomez, Naman Goyal, Somya Jain, Douwe Kiela, Tristan Thrush, Francisco Guzmán Facebook AI {guw,vishrav,angelafan,sahir}@fb.com {namangoyal,somyaj,dkiela,tthrush,fguzman}@fb.com Abstract We present the results of the first task on Large- Scale Multilingual Machine Translation. The task consists on the many-to-many evaluation of a single model across a variety of source and target languages. This year, the task con- sisted on three different settings: (i) SMALL- TASK1 (Central/South-Eastern European Lan- guages), (ii) the SMALL-TASK2 (South East Asian Languages), and (iii) FULL-TASK (all 101 x 100 language pairs). All the tasks used the FLORES-101 datasetas the evaluation benchmark. To ensure the longevity of the dataset, the test sets were not publicly released and the models were evaluated in a controlled environment on Dynabench.There were a total of 10 participating teams for the tasks, with a total of 151 intermediate model submissions and 13 final models. This year’s result show a significant improvement over the known base- lines with +17.8 BLEU for SMALL-TASK2, +10.6 for FULL-TASK and +3.6 for SMALL- TASK1. 1 Introduction Despite recent advances in translation quality for a handful of languages and domains, MT systems still perform poorly on low-resource languages. Yet, most of the world’s population speak low- resource languages and would benefit from im- provements in translation quality on their native languages. As a result, the field has been shift- ing focus towards the evaluation of MT in low- resource situations (Thu et al., 2016; Guzmán et al., 2019; Barrault et al., 2020; ∀ et al., 2020; Ebrahimi et al., 2021; Kuwanto et al., 2021). However, these efforts have had poor coverage of low-resource languages which limits our understanding on gen- eralization.More importantly, there has been little focus on the evaluation of true many-to-many mul- tilingual models, which hampers the progress of the field despite all the recent excitement on this research direction (Fan et al., 2020). The recent release of the FLORES-101 (Goyal et al., 2021) benchmark made possible to evaluate massively multilingual systems in a consistent way. The benchmark consists of 3001 sentences sam- pled from English Wikipedia and professionally translated in 101 languages. This poses a unique opportunity to understand translation across many languages with varied typology, resources, etc. In this first multilingual large-scale shared task, we use the FLORES-101 benchmark to evaluate the progress on massively multilingual translation, where the evaluation is performed in a non-English- centric way. We propose 3 different tasks: two small tasks involving translation between 6 lan- guages each (30 pairs), and a large task involving the translation across 101 languages (10K pairs). In the remainder of this paper, we describe the task setup, the participants, and the official results for the task. We also analyze the results to understand better the languages for which progress has been attained, and those where a gap in …

Content truncated to 500 words