A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3053 - 3070 July 10-15, 2022 ©2022 Association for Computational Linguistics A Few Thousand Translations Go A Long Way! Leveraging Pre-trained Models for African News Translation David Ifeoluwa Adelani1∗, Jesujoba Oluwadara Alabi2∗, Angela Fan3∗, Julia Kreutzer4∗, Xiaoyu Shen5, Machel Reid6∗, Dana Ruiter1, Dietrich Klakow1, Peter Nabende7∗, Ernie Chang1∗, Tajuddeen R. Gwadabe8∗, Freshia Sackey9∗, Bonaventure F. P. Dossou10∗, Chris Chinenye Emezue11∗, Colin Leong12∗, Michael Beukman13∗, Shamsuddeen H. Muhammad14∗, Guyo D. Jarso∗, Oreen Yousuf15∗, Andre N. Rubungo16∗, Gilles Hacheme17∗, Eric P. Wairagala7∗, Muhammad U. Nasir18∗, Benjamin A. Ajibade∗, Tunde Oluwaseyi Ajayi∗, Yvonne Wambui Gitau∗, Jade Abbott∗, Mohamed Ahmed19∗, Millicent Ochieng19∗, Anuoluwapo Aremu∗, Perez Ogayo20∗, Jonathan Mukiibi7∗, Fatoumata Ouoba Kabore∗, Godson Koffi Kalipe∗, Derguene Mbaye21∗ Allahsera Auguste Tapo22∗, Victoire M. Koagne∗, Edwin Munkoh-Buabeng∗, Valencia Wagner23∗, Idris Abdulmumin24∗, Ayodele Awokoya25∗, Happy Buzaaba∗, Blessing Sibanda26∗, Andiswa Bukula27∗, Sam Manthalu28 ∗Masakhane NLP, 1Saarland University, Germany, 2Inria, France, 3Meta AI, 4Google Research, 5Amazon Alexa AI, 6The University of Tokyo, Japan, 7Makerere University, Kampala, Uganda, 8UCAS, China, 9 JKUAT, Kenya, 10Jacobs University, Germany, 11TUM, Germany, 12University of Dayton, USA, 13University of the Witwatersrand, South Africa, 14LIAAD-INESC TEC, Porto, Portugal, 15Uppsala University, Sweden, 16UPC, Spain, 17Ai4Innov 18Ominor AI 19Microsoft Africa Research Institute, Kenya 20CMU, USA, 21Baamtu, 22RIT, USA, 23SPU, South Africa, 24ABU, Nigeria, 25UI Ibadan, Nigeria, 26NUST, Namibia 27SADiLaR, South Africa, 28University of Malawi, Malawi Abstract Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally cho- sen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation sys- tems for 16 African languages. We focus on two questions: 1) How can pre-trained mod- els be used for languages not included in the initial pre-training? and 2) How can the re- sulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus cover- ing 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high- quality translation data. 1 Introduction Enormous efforts have been invested in making language and translation models more multilingual while leveraging the maximal amount of data for training, most prominently large crawls of mono- lingual and parallel data from the web (El-Kishky et al., 2020; Schwenk et al., 2021b,a; Xue et al., 2021b). The resulting models are now capable of translating between hundreds of languages, includ- ing language pairs that in isolation do not have large collections of parallel data (Tang et al., 2020; Xue …

Content truncated to 500 words