DENTRA: Denoising and Translation Pre-training for Multilingual Machine Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1057 - 1067 December 7-8, 2022 ©2022 Association for Computational Linguistics DENTRA: Denoising and Translation Pre-training for Multilingual Machine Translation Samta Kamboj, Sunil Kumar Sahu, Neha Sengupta Inception Institute of Artificial Intelligence Abu Dhabi, UAE samta.kamboj@g42.ai, {sunil.sahu, neha.sengupta}@inceptioniai.org Abstract In this paper, we describe our submission to the WMT-2022: Large-Scale Machine Transla- tion Evaluation for African Languages under the Constrained Translation track. We intro- duce DENTRA, a novel pre-training strategy for a multilingual sequence-to-sequence trans- former model. DENTRA pre-training combines denoising and translation objectives to incor- porate both monolingual and bitext corpora in 24 African, English, and French languages. To evaluate the quality of DENTRA, we fine-tuned it with two multilingual machine translation configurations, one-to-many and many-to-one. In both pre-training and fine-tuning, we em- ploy only the datasets provided by the organ- isers. We compare DENTRA against a strong baseline, M2M-100, in different African multi- lingual machine translation scenarios and show gains in 3 out of 4 subtasks. 1 Introduction Despite the compelling performance of machine translation (MT) in many European and Asian lan- guages, their quality in African languages is rela- tively low. This is primarily because there are ap- proximately 2000 known languages in the African continent, out of which very few languages have any significant presence on the Web (Eberhard et al., 2020; Emezue and Dossou, 2021; Adelani et al., 2022a). As a result, many African languages are not included in publicly available bitext re- sources, which are typically created by employing heuristics on large amounts of data crawled from the Web (Tiedemann, 2012; El-Kishky et al., 2020; Schwenk et al., 2021; Goyal et al., 2022). To take a step towards addressing the under- representation of African languages in MT, WMT- 2022 presented the Constrained Translation track under Large-Scale Multilingual African Transla- tion (Adelani et al., 2022b), which releases bitext and monolingual corpora for 24 African languages, and participants are only allowed to use the pro- vided data. Our submission is to the aforemen- tioned track. Roughly 34% of the provided data is monolin- gual, spread across 24 African languages pertain- ing to this task. Since the volume of bitext data provided is limited, our submission aims to lever- age the monolingual data to improve the perfor- mance of a multilingual machine translation model. To leverage monolingual data in translation, pre- training the model is an obvious choice. There are several existing multilingual pre- trained models such as mBART (Liu et al., 2020), mT5 (Xue et al., 2021), byT5 (Xue et al., 2022), mRASP (Pan et al., 2021), mRASP2 (Pan et al., 2021), and M2M-1001 (Fan et al., 2021) that are trained on monolingual data, bitext data, or both, and have been demonstrated to improve transla- tion performance for specific language pairs. But, these models do not include many of the African languages of interest. For example, the 50 lan- guages covered by mBART50 include only two out of the 24 African languages in the shared task …

Content truncated to 500 words