AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1306–1320 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1306 AFROMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages Machel Reid1, Junjie Hu2, Graham Neubig2, Yutaka Matsuo1 1The University of Tokyo, 2Carnegie Mellon University {machelreid,matsuo}@weblab.t.u-tokyo.ac.jp {junjieh,gneubig}@cs.cmu.edu Abstract Reproducible benchmarks are crucial in driv- ing progress of machine translation research. However, existing machine translation bench- marks have been mostly limited to high- resource or well-represented languages. De- spite an increasing interest in low-resource ma- chine translation, there are no standardized reproducible benchmarks for many African languages, many of which are used by mil- lions of speakers but have less digitized tex- tual data. To tackle these challenges, we pro- pose AFROMT, a standardized, clean, and reproducible machine translation benchmark for eight widely spoken African languages. We also develop a suite of analysis tools for system diagnosis taking into account unique properties of these languages. Furthermore, we explore the newly considered case of low-resource focused pretraining and develop two novel data augmentation-based strategies, leveraging word-level alignment information and pseudo-monolingual data for pretrain- ing multilingual sequence-to-sequence mod- els. We demonstrate significant improvements when pretraining on 11 languages, with gains of up to 2 BLEU points over strong base- lines. We also show gains of up to 12 BLEU points over cross-lingual transfer baselines in data-constrained scenarios. All code and pre- trained models will be released as further steps towards larger reproducible benchmarks for African languages.1 1 Introduction Accuracy of machine translation systems in many languages has improved greatly over the past sev- eral years due to the introduction of neural ma- chine translation (NMT) techniques (Bahdanau et al., 2015; Sutskever et al., 2014; Vaswani et al., 2017), as well as scaling to larger models (Ott et al., 2018). However, many of these advances have 1Source code, pretrained models, and data can be found at https://github.com/machelreid/afromt been demonstrated in settings where very large parallel datasets are available (Meng et al., 2019; Arivazhagan et al., 2019), and NMT systems of- ten underperform in low-resource settings when given small amounts of parallel corpora (Koehn and Knowles, 2017; Guzmán et al., 2019). One solution to this has been leveraging multilingual pretraining on large sets of monolingual data (Con- neau and Lample, 2019; Song et al., 2019; Liu et al., 2020), leading to improvements even with smaller parallel corpora. However, this thread of work has focused on scenarios with the following two properties: (1) pretraining on a plurality of European languages and (2) cases in which the monolingual pretraining data greatly exceeds the parallel data used for finetuning (often by over 100 times) (Guzmán et al., 2019; Liu et al., 2020). However, in the case of many languages in the world, the above two properties are often not satis- fied. In particular, taking the example of African languages (the focus of our work), existing (small) parallel corpora for English-to-African language pairs often comprise the majority of available monolingual data in the corresponding …

Content truncated to 500 words