Findings of the WMT’22 Shared Task on Large-Scale Machine Translation Evaluation for African Languages

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 773 - 800 December 7-8, 2022 ©2022 Association for Computational Linguistics Findings of the WMT’22 Shared Task on Large-Scale Machine Translation Evaluation for African Languages David Ifeoluwa Adelani♠ Md Mahfuz Ibn Alam† Antonios Anastasopoulos† Akshita Bhagia♢ Marta R. Costa-jussà♡ Jesse Dodge♢ Fahim Faisal† Natalia Fedorova‡ Christian Federmann⊗ Francisco Guzmán♡ Sergey Koshelev‡ Jean Maillard♡ Vukosi Marivate# Jonathan Mbuya† Alexandre Mourachko♡ Safiyyah Saleem♡ Holger Schwenk♡ Guillaume Wenzek♡ †George Mason University ♠University College London ♡Meta AI ⊗Microsoft Research ‡Toloka #University of Pretoria ♢AI2 Abstract We present the results of the WMT’22 Shared Task on Large-Scale Machine Translation Eval- uation for African Languages. The shared task included both a data and a systems track, along with additional innovations, such as a focus on African languages and extensive human eval- uation of submitted systems. We received 14 system submissions from 8 teams, as well as 6 data track contributions. We report a large progress in the quality of translation for African languages since the last iteration of this shared task: there is an increase of about 7.5 BLEU points across 72 language pairs, and the aver- age BLEU scores went from 15.09 to 22.60. 1 Introduction A large portion of the world’s population speak low-resource languages, and would benefit from improvements in translation quality on their na- tive languages. Recent advances in translation quality, particularly from massively multilingual models (Fan et al.; Ma et al., 2021), have enabled progress in the translation quality of low-resource languages. However, many languages have seen little to no progress. This is particularly true for African languages. For example, in the 2021 Large Scale Multilingual Evaluation shared task (Wenzek et al., 2021), there was a progress2 of +19.3 avg. BLEU when translating into languages like Irish and Welsh (included in the Other Indo-European grouping), but only a progress of +3.5 avg. BLEU when translating into languages like Fula and Igbo (included in the Nilotic/Atlantic Congo grouping). The African continent is home to a rich diversity of languages. Around a third of the world’s living languages are from Africa and only a small fraction Author names are sorted in alphabetical order. 2Progress was determined by comparing the average BLEU score between the best system in the task vs. the baseline. of the resources in NLP and Machine translation are dedicated to them (Orife et al., 2020). As a re- sult, African language speakers do not benefit from language technologies similarly to other Global North language speakers (Blasi et al., 2022). A ma- jor (but not the sole) hurdle to building language technologies for African languages is data avail- ability (Joshi et al., 2021), with significant efforts underway stemming –importantly– from Africa it- self (Nekoto et al., 2020). To bring attention of the research community to the challenges of translating African Languages, this year we focus on the multilingual evaluation of 24 African Languages together with French and English. We base our evaluation on the benchmark provided by FLORES (Goyal et al., …

Content truncated to 500 words