The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation Naman Goyal1, Cynthia Gao1, Vishrav Chaudhary1, Peng-Jen Chen1, Guillaume Wenzek2, Da Ju1, Sanjana Krishnan1, Marc’Aurelio Ranzato1, Francisco Guzm´an1, Angela Fan2,3 1Facebook AI Research, USA, 2Facebook AI Research, France, 3LORIA flores@fb.com Abstract One of the biggest challenges hindering pro- gress in low-resource and multilingual ma- chine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource lan- guages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation bench- mark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sen- tences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset en- ables better assessment of model quality on the long tail of low-resource languages, in- cluding the evaluation of many-to-many multi- lingual translation systems, as all translations are fully aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine trans- lation community and beyond. 1 Introduction Machine translation (MT) is one of the most successful applications in natural language pro- cessing, as exemplified by its numerous practical applications and the number of contributions on this topic at major machine learning and nat- ural language processing venues. Despite recent advances in translation quality for a handful of lan- guage pairs and domains, MT systems still perform poorly on low-resource languages, that is, lan- guages without a lot of training data. In fact, many low-resource languages are not even supported by most popular translation engines. Yet, much of the world’s population speak low-resource lan- guages and would benefit from improvements in translation quality on their native languages. As a result, the field has been increasing focus towards low-resource languages. At present, there are very few benchmarks on low-resource languages. These often have very low coverage of low-resource languages (Riza et al., 2016; Thu et al., 2016; Guzm´an et al., 2019; Barrault et al., 2020a; Nekoto et al., 2020; Ebrahimi et al., 2021; Kuwanto et al., 2021), lim- iting our understanding of how well methods gen- eralize and scale to a larger number of languages with a diversity of linguistic features. There are some benchmarks that have high coverage, but these are often in specific domains, like COVID- 19 (Anastasopoulos et al., 2020) or religious texts (Christodouloupoulos and Steedman, 2015; Malaviya et al., 2017; Tiedemann, 2018; Agi´c and Vuli´c, 2019); or have low quality because they are built using automatic approaches (Zhang et al., 2020; Schwenk et al., 2019, 2021). As a result, it is difficult to draw firm conclusions about research efforts on low-resource MT. In particular, there are even fewer benchmarks that are suitable for evaluation of many-to-many multilingual transla- tion, as these require multi-lingual alignment (i.e., having the translation of the same sentence in multiple languages), which hampers the progress of the field despite all the …

Content truncated to 500 words