Domain Curricula for Code-Switched MT at MixMT 2022

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1162 - 1169 December 7-8, 2022 ©2022 Association for Computational Linguistics Domain Curricula for Code-Switched MT at MixMT 2022 Lekan Raheem & Maab Elrashid African Institute for Mathematical Sciences (AIMS) {rwaliyu,mnimir}@aimsammi.org Abstract In multilingual colloquial settings, it is a habit- ual occurrence to compose expressions of text or speech containing tokens or phrases of differ- ent languages, a phenomenon popularly known as code-switching or code-mixing (CMX). We present our approach and results for the Code- mixed Machine Translation (MixMT) shared task at WMT 2022: the task consists of two subtasks, monolingual to code-mixed ma- chine translation (Subtask-1) and code-mixed to monolingual machine translation (Subtask- 2). Most non-synthetic code-mixed data are from social media but gathering a significant amount of this kind of data would be laborious and this form of data has more writing variation than other domains, so for both subtasks, we experimented with data schedules for out-of- domain data. We jointly learn multiple domains of text by pretraining and fine-tuning, com- bined with a sentence alignment objective. We found that switching between domains caused improved performance in the domains seen ear- liest during training, but depleted the perfor- mance on the remaining domains. A contin- uous training run with strategically dispensed data of different domains showed a significantly improved performance over fine-tuning. 1 Introduction Code-mixing (CMX) denotes the alternation of two languages within a single utterance (Poplack, 1980; Sitaram et al., 2019). Code-mixing occurs mostly in unofficial groups in multilingual environments. More than 77% of Asians are multilingual (Ra- makrishnan and Ahmad, 2014), and other statistics estimate that 64.5% of Europeans speak more than two languages, with more than 80% of adults in the region being bilingual (Eurostat, 2019). Code- mixing happens far more often in conversations than in writing, and mostly in unofficial settings, hence it rarely occurs in documented settings. This makes substantial data gathering for computational approaches to translations of code-mixed language difficult. Parallel corpora for code-switched data is very scarce (Menacer et al., 2019), this is because code-mixing mostly occurs in unofficial conversa- tions like social media interactions. Contemporary Neural Machine Translation (NMT) mostly makes use of parametric sequence- to-sequence models (Bahdanau et al., 2014; Vaswani et al., 2017), where an encoder receives a source sentence and outputs a set of hidden states, the decoder then scrutinizes these hidden states at each step, and outputs a sequence of softmax distribution over the target vocabulary space. Con- sidering that we would need vast quantities of data to train an adequate NMT for this task, we leverage large-scale synthetic and available small data and notably rank data on domain relevance, by fine- tuning with it, initiating training with the relevant domain and strategically placing it at the premier batches of the training data. Essentially, the characteristics of the data an NMT model is trained on are paramount to its trans- lation quality, in particular in terms of size and domain. It is quintessential to train NMT …

Content truncated to 500 words