AdvAug: Robust Adversarial Augmentation for Neural Machine Translation

AdvAug: Robust Adversarial Augmentation for Neural Machine Translation Yong Cheng, Lu Jiang, Wolfgang Macherey and Jacob Eisenstein Google Research {chengyong, lujiang, wmach, jeisenstein}@google.com Abstract In this paper, we propose a new adver- sarial augmentation method for Neural Ma- chine Translation (NMT). The main idea is to minimize the vicinal risk over virtual sen- tences sampled from two vicinity distributions, of which the crucial one is a novel vicin- ity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sen- tence pairs. We then discuss our approach, AdvAug, to train NMT models using the em- beddings of virtual sentences in sequence-to- sequence learning. Experiments on Chinese- English, English-French, and English-German translation benchmarks show that AdvAug achieves significant improvements over the Transformer (up to 4.9 BLEU points), and sub- stantially outperforms other data augmentation techniques (e.g. back-translation) without us- ing extra corpora. 1 Introduction Recent work in neural machine translation (Bah- danau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) has led to dramatic improvements in both research and commercial systems (Wu et al., 2016). However, a key weakness of contempo- rary systems is that performance can drop dra- matically when they are exposed to input pertur- bations (Belinkov and Bisk, 2018; Cheng et al., 2019), even when these perturbations are not strong enough to alter the meaning of the input sentence. Consider a Chinese sentence, “zhejia feiji meiyou zhuangshang zhujia huo yiyuan, shizai shi qiji”. If we change the word “huo (或)” to its syn- onym“ji (及)”, the Transformer model will gen- erate contradictory results of “It was indeed a mira- cle that the plane did not touch down at home or hospital.” versus “It was a miracle that the plane landed at home and hospital.” Such perturbations can readily be found in many public benchmarks and real-world applications. This lack of stability not only lowers translation quality but also inhibits applications in more sensitive scenarios. At the root of this problem are two interrelated issues: first, machine translation training sets are insufficiently diverse, and second, NMT architec- tures are powerful enough to overfit — and, in extreme cases, memorize — the observed train- ing examples, without learning to generalize to unseen perturbed examples. One potential solu- tion is data augmentation which introduces noise to make the NMT model training more robust. In general, two types of noise can be distinguished: (1) continuous noise which is modeled as a real- valued vector applied to word embeddings (Miyato et al., 2016, 2017; Cheng et al., 2018; Sato et al., 2019), and (2) discrete noise which adds, deletes, and/or replaces characters or words in the observed sentences (Belinkov and Bisk, 2018; Sperber et al., 2017; Ebrahimi et al., 2018; Michel et al., 2019; Cheng et al., 2019; Karpukhin et al., 2019). In both cases, the challenge is to ensure that the noisy examples are still semantically valid translation pairs. In the case of continuous noise, it only en- sures that the noise vector lies …

Content truncated to 500 words