Understanding Data Augmentation in Neural Machine Translation: Two Perspectives towards Generalization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5689–5695, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5689 Understanding Data Augmentation in Neural Machine Translation: Two Perspectives towards Generalization Guanlin Li✏ ⇤, Lemao Liuλ, Guoping Huangλ, Conghui Zhu✏ , Tiejun Zhao✏, Shuming Shiλ ✏Harbin Institute of Technology, λ Tencent AI Lab {epsilonlee.green}@gmail.com, {chzhu, tjzhao}@hit.edu.cn, {redmondliu, donkeyhuang, shumingshi}@tencent.com Abstract Many Data Augmentation (DA) methods have been proposed for neural machine translation. Existing works measure the superiority of DA methods in terms of their performance on a specific test set, but we find that some DA methods do not exhibit consistent improve- ments across translation tasks. Based on the observation, this paper makes an initial at- tempt to answer a fundamental question: what benefits, which are consistent across different methods and tasks, does DA in general obtain? Inspired by recent theoretic advances in deep learning, the paper understands DA from two perspectives towards the generalization ability of a model: input sensitivity and prediction margin, which are defined independent of spe- cific test set thereby may lead to findings with relatively low variance. Extensive experiments show that relatively consistent benefits across five DA methods and four translation tasks are achieved regarding both perspectives. 1 Introduction Data Augmentation (DA) is a training paradigm that has been proved to be very effective in many modalities (Park et al., 2019; Perez and Wang, 2017; Sennrich et al., 2016a), especially for classi- fication (Perez and Wang, 2017). In structured do- main, Neural Machine Translation (NMT) is the frontier of DA research (Sennrich et al., 2016a; Norouzi et al., 2016; Zhang and Zong, 2016; Fadaee et al., 2017; Wang et al., 2018; Zhang et al., 2019; Edunov et al., 2018; Fadaee and Monz, 2018). However, by investigating a vari- ety of DA methods, we find that their test per- formance across different translation tasks does not exhibit consistent improvement, and this phe- nomenon can be initially observed in (Wang et al., 2018) as well. The reason might be the evaluation ⇤ Work done at Tencent AI Lab. metric on a specific test set when compared to the whole data population, which generates all possi- ble data, has large variance so that leads to the in- consistency. This evaluation dilemma is also rec- ognized and explored by Recht et al. (2018, 2019); Werpachowski et al. (2019), and is especially no- torious for language generation tasks (Chaganty et al., 2018; Hashimoto et al., 2019) where the evaluation metrics, e.g. BLEU (Papineni et al., 2001), are extrinsic and heavily relies on the ref- erence provided. Therefore, we ask a fundamental question: what benefits, which are more consistent across different DA methods and translation tasks, can DA in general obtain? A direct answer to the above question is to use generalization gap (Kawaguchi et al., 2018) de- fined by the difference between population risk and empirical risk. This measure does not rely on any …

Content truncated to 500 words