Auxiliary Subword Segmentations as Related Languages for Low Resource Multilingual Translation

Auxiliary Subword Segmentations as Related Languages for Low Resource Multilingual Translation Nishant Kambhatla Logan Born School of Computing ScienceAnoop Sarkar Simon Fraser University 8888 University Drive, Burnaby BC, Canada {nkambhat, loborn, anoop}@sfu.ca Abstract We propose a novel technique of combin- ing multiple subword tokenizations of a single source-target language pair for use with multilingual neural translation train- ing methods. These alternate segmenta- tions function like related languages in multilingual translation, improving trans- lation accuracy for low-resource languages and producing translations that are lex- ically diverse and morphologically rich. We also introduce a cross-teaching tech- nique which yields further improvements in translation accuracy and cross-lingual transfer between high- and low-resource language pairs. Compared to other strong multilingual baselines, our approach yields average gains of +1.7 BLEU across the four low-resource datasets from the multi- lingual TED-talks dataset. Our technique does not require additional training data and is a drop-in improvement for any ex- isting neural translation system. 1 Introduction Multilingual neural machine translation (NMT, Dong et al. 2015; Johnson et al. 2017) models are capable of translating from multiple source and target languages. Besides allowing efficient pa- rameter sharing (Aharoni et al., 2019) these mod- els facilitate inherent transfer learning (Zoph et al., 2016; Firat et al., 2016) that can especially bene- fit low resource languages (Nguyen and Chiang, 2017; Gu et al., 2018; Neubig and Hu, 2018; © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. Tan et al., 2019). A common technique to ad- dress lexical sharing and complex morphology in multilingual NMT is to decompose longer words into shorter subword units (Sennrich et al., 2016). Since subword units are produced using heuris- tic methods, not all subwords are created equally. This can put low- and extremely low-resource lan- guages at a disadvantage, even when these lan- guages are paired with a suitable high resource lan- guage. To diminish the impact of rare subwords in NMT, Kambhatla et al. (2022) leverage cipher- texts to augment the training data by constructing multiple-views of the source text. “Soft” decom- position methods based on transfer learning (Wang et al., 2018) address the problem of sub-optimal word segmentation with shared character-level lex- ical and sentence representations across multiple source languages (Gu et al., 2018). Wang et al. (2021) addressed this problem with a multiview- subword regularization technique that also im- proves the effectiveness of cross-lingual transfer in pretrained multilingual representations by si- multaneously finetuning on different input seg- mentations from a heuristic and a probabilistic to- kenizer. While subword-regularization methods (Kudo, 2018; Provilkov et al., 2020) have been widely explored in NMT, this work is the first to study them together with multilingual training methods. Concretely, we construct pairs of “related lan- guages” by segmenting an input corpus twice, each time with a different vocabulary size and algorithm for finding subwords; we use these “languages” (really, views of the same language) for multi- lingual training of an NMT model. We propose Multi-Sub …

Content truncated to 500 words