Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution

Proceedings of the 18th International Conference on Spoken Language Translation, pages 287–293 Bangkok, Thailand (Online), August 5–6, 2021. ©2021 Association for Computational Linguistics 287 Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution Toan Q. Nguyen University of Notre Dame tnguye28@nd.edu Kenton Murray Johns Hopkins University kenton@jhu.edu David Chiang University of Notre Dame dchiang@nd.edu Abstract In this paper, we investigate the driving factors behind concatenation, a simple but effective data augmentation method for low-resource neural machine translation. Our experiments suggest that discourse context is unlikely the cause for concatenation improving BLEU by about +1 across four language pairs. Instead, we demonstrate that the improvement comes from three other factors unrelated to discourse: context diversity, length diversity, and (to a lesser extent) position shifting. 1 Introduction Many attempts have been made to augment neural machine translation (MT) systems to use discourse context (Junczys-Dowmunt, 2019; Stojanovski and Fraser, 2019; Saunders et al., 2020; Zhang et al., 2018; Sun et al., 2020; L¨aubli et al., 2018; Kim et al., 2019; Tan et al., 2019; Zheng et al., 2020; Jean et al., 2017). One particularly simple method is to concatenate consecutive pairs of sentence- pairs during training, but not during translation (Agrawal et al., 2018; Tiedemann and Scherrer, 2017; Ngo and Trinh, 2021; Kondo et al., 2021).1 In this paper, we confirm that this simple method helps, by roughly +1 BLEU across four low- resource language pairs. But we demonstrate that the reason it helps is not discourse context, be- cause concatenating random pairs of sentence-pairs yields the same improvement. Instead, we view concatenation as a kind of data augmentation or noising method (one which pleasantly requires no alteration to the text, unlike data augmentation methods that disturb word order 1As this paper was being finalized, Kondo et al. (2021) published independent work also presenting random concate- nation as data augmentation for NMT. They find that concate- nation helps the model translate long sentences better, while the focus of the present paper is to explain thoroughly why it helps. (Belinkov and Bisk, 2018; Anastasopoulos et al., 2019) or replace words with automatically-selected words (Gao et al., 2019; Fadaee et al., 2017; Wang et al., 2018)). Concatenating random sentences is easier than concatenating consecutive sentences, because many parallel corpora discard document boundaries, drop sentence-pairs, or even reorder sentence-pairs, so it can be difficult to know which sentence-pairs are truly consecutive. But the fact that random concatenation helps so much creates a mystery, which is the focus of the paper. If the reason is not discourse context, what is the reason? We consider three new hypotheses: • Random concatenation creates greater diver- sity of positions, because it lets the model see sentences shifted by effectively random distances. • Random concatenation creates greater diver- sity of contexts, helping the model learn what not to attend to. • Random concatenation creates greater diver- sity of sentence lengths within a minibatch. Through a careful ablation study, we demonstrate that all three of these factors more or …

Content truncated to 500 words