Penalizing Divergence: Multi-Parallel Translation for Low-Resource Languages of North America

Proceedings of the 29th International Conference on Computational Linguistics, pages 4292–4298 October 12–17, 2022. 4292 Penalizing Divergence: Multi-Parallel Translation for Low-Resource Languages of North America Garrett Nicolai Changbing Yang Miikka Silfverberg University of British Columbia first.last@ubc.ca Abstract This paper explores a special case in multi- lingual machine translation: so called multi- parallel translation, where the target data for all language pairs are identical. While multi- parallelism offers benefits which are not avail- able in a standard translation setting, transla- tion models can easily overfit when training data are limited. We introduce a regularizer, the divergence penalty, which penalizes the translation model when it represents source sentences with identical target translations in divergent ways. Experiments on very low- resourced Indigenous North American lan- guages show that an initially deficient multi- lingual translator can improve by 4.9 BLEU through mBART pre-training, and 5.5 BLEU points with the strategic addition of mono- lingual data, and that a divergence penalty leads to further increases of 0.4 BLEU. Further experiments on Germanic languages demon- strate a improvement of 0.5 BLEU when apply- ing the divergence penalty. An investigation of the neural encoder representations learned by our translation models shows that the di- vergence penalty encourages models to learn a unified neural interlingua. 1 Introduction Bilingual neural translation models typically re- quire millions of parallel sentences to achieve ad- equate quality. A vast majority of the world’s lan- guages lack sufficient parallel corpora, and transla- tion efforts for low-resource languages have turned to multilingual methods, leveraging linguistic sim- ilarity to augment a deficient signal with plenti- ful, albeit sometimes noisy, data from related lan- guages (Aharoni et al., 2019; Goyal et al., 2020). In this paper, we explore a very specific multilin- gual translation setting: multi-parallel translation. Here, models are trained on documents, such as the proceedings of the European Parliament, col- lections of subtitles, and the Bible. Each sentence has translations in many languages, providing not just a bilingual signal, but one that is bilingual in many directions. We explore a massively multi-parallel docu- ment: the Bible, which has hundreds of translations. While it represents a very restricted domain, the Bible is the only parallel document available for many languages and multi-parallel translation is, therefore, of key importance for low-resource NLP. Earlier work shows that multi-parallel translation systems can in practice deliver poor results when available training data for individual languages are very small (Mueller et al., 2020). In this setting, the performance of the translation models degrades when the number of source languages is increased. Adapting pre-trained multilingual models such as mBART has also not led to significant progress (Lee et al., 2022). For languages that are not closely-related to the languages in the model, high- quality translation remains an unsatisfied goal. We hypothesize that this drop in performance is a consequence of an inability of the model to learn a neural interlingua (Johnson et al., 2017), that is, a shared semantic representation for source lan- guages. This prevents knowledge transfer between languages, degrading …

Content truncated to 500 words