Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8502–8516 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8502 Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach Víctor M. Sánchez-Cartagena, Miquel Esplà-Gomis Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez Dep. de Llenguatges i Sistemes Informàtics Universitat d’Alacant E-03690 Sant Vicent del Raspeig (Spain) {vmsanchez,mespla,japerez,fsanchez}@dlsi.ua.es Abstract In the context of neural machine translation, data augmentation (DA) techniques may be used for generating additional training sam- ples when the available parallel data are scarce. Many DA approaches aim at expanding the support of the empirical data distribution by generating new sentence pairs that contain in- frequent words, thus making it closer to the true data distribution of parallel sentences. In this paper, we propose to follow a completely different approach and present a multi-task DA approach in which we generate new sentence pairs with transformations, such as reversing the order of the target sentence, which pro- duce unfluent target sentences. During train- ing, these augmented sentences are used as auxiliary tasks in a multi-task framework with the aim of providing new contexts where the target prefix is not informative enough to pre- dict the next word. This strengthens the en- coder and forces the decoder to pay more at- tention to the source representations of the en- coder. Experiments carried out on six low- resource translation tasks show consistent im- provements over the baseline and over DA methods aiming at extending the support of the empirical data distribution. The systems trained with our approach rely more on the source tokens, are more robust against domain shift and suffer less hallucinations. 1 Introduction In order to train reliable neural machine transla- tion (NMT) systems, large amounts of parallel sen- tences —sentence pairs in two languages that are mutual translations— are needed, which constitutes a critical barrier for low-resource language pairs. This problem has been addressed through different approaches, such as transfer-learning from high- resource language pairs (Kocmi and Bojar, 2018), using linguistic annotations (Sennrich and Had- dow, 2016), training multilingual systems (John- son et al., 2017) and applying data augmentation strategies (Li et al., 2019; Feng et al., 2021), i.e., artificially generating additional parallel sentences. Data augmentation (DA) is formalised by many authors as a solution to a data distribution mismatch problem (Wang et al., 2018; Wei et al., 2020). The data distribution of the sentence pairs observed in the training corpus, ˆp(x, y), differs from the true data distribution, p(x, y). Hence, the system should be trained on a training set that follows q(x, y), an augmented version of ˆp(x, y) with a wider support. In this way, the trained system is less likely to face totally out-of-distribution data when translating. In this paper, we propose a completely different framework for DA in which we generate additional parallel sentences which, despite being completely unlikely under the data distribution, systematically improve the quality of the resulting NMT system. Inspired by one-to-many multilingual NMT, where richer encoder representations …

Content truncated to 500 words