Exploring Enhanced Code-Switched Noising for Pretraining in Neural Machine Translation

Findings of the Association for Computational Linguistics: EACL 2023, pages 984–998 May 2-6, 2023 ©2023 Association for Computational Linguistics Exploring Enhanced Code-Switched Noising for Pretraining in Neural Machine Translation Vivek Iyer School of Informatics University of Edinburgh v.iyer@sms.ed.ac.uk Arturo Oncevay School of Informatics University of Edinburgh aoncevay@ed.ac.uk Alexandra Birch School of Informatics University of Edinburgh a.birch@ed.ac.uk Abstract Multilingual pretraining approaches in Neu- ral Machine Translation (NMT) have shown that training models to denoise synthetic code- switched data can yield impressive performance gains — owing to better multilingual semantic representations and transfer learning. However, they generated the synthetic code-switched data using non-contextual, one-to-one word transla- tions obtained from lexicons - which can lead to significant noise in a variety of cases, in- cluding the poor handling of polysemes and multi-word expressions, violation of linguistic agreement and inability to scale to agglutinative languages. To overcome these limitations, we propose an approach called Contextual Code- Switching (CCS), where contextual, many-to- many word translations are generated using a `base' NMT model. We conduct experiments on 3 different language families - Romance, Uralic, and Indo-Aryan - and show significant improve- ments (by up to 5.5 spBLEU points) over the previous lexicon-based SOTA approaches. We also observe that small CCS models can per- form comparably or better than massive models like mBART50 and mRASP2, depending on the size of data provided. Lastly, through ablation studies, we highlight the major code-switching aspects (including context, many-to-many sub- stitutions, code-switching language count etc.) that contribute to the enhanced pretraining of multilingual NMT models. 1 Introduction Recent research in Neural Machine Translation (NMT) has focused on the pretraining of massively multilingual models (Aharoni et al., 2019; Siddhant et al., 2022; Costa-jussà et al., 2022) - due to their high scalability, easy deployability and state-of-the- art (SOTA) performances (Tran et al., 2021; Yang et al., 2021). One of the most common pretrain- ing approaches trains a model to reconstruct (or ``denoise") a sentence noised using one or more mechanisms. Following masking (Song et al., 2019; (a) Example sentences (Pan et al., 2021, Figure. 6) (b) Translation Errors in the noised sentence generated using AA. Parentheses indicate code-switching language. Figure 1: Example of the errors induced by Aligned Augmentation (AA). GT refers to `Ground Truth' trans- lations, as provided by native speakers. Lewis et al., 2020), synthetic code-switching1 has emerged as a more effective noising mechanism (Yang et al., 2020b; Pan et al., 2021). With the mo- tive of moving the denoising task from language modeling to machine translation, these works pro- pose to randomly code-switch input sentences, and then to train the MT model to denoise these sen- tences to the original monolingual ones. The most noteworthy system in this line of research is the massively multilingual model mRASP2 (Pan et al., 2021). It was pretrained using an algorithm called Aligned Augmentation (AA) - that constructs a synthetic code-switched sentence using dictionary-based word-level translations, fol- lowed by contrastive learning to semantically align this sentence with the reference sentence. Pan et al. …

Content truncated to 500 words