Manifold Adversarial Augmentation for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3184–3189 August 1–6, 2021. ©2021 Association for Computational Linguistics 3184 Manifold Adversarial Augmentation for Neural Machine Translation Guandan Chen, Kai Fan, Kaibo Zhang, Boxing Chen, Zhongqiang Huang Alibaba DAMO Academy {guandan.cgd, k.fan, kaibo.zkb}@alibaba-inc.com {boxing.cbx, z.huang}@alibaba-inc.com Abstract Improving the robustness of neural machine translation models on variations of input sen- tences is an active area of research. In this paper, we propose a simple data augmenta- tion approach by sampling virtual sentences from the vicinity distributions in higher-level representations, constructed either from indi- vidual training samples via adversarial learn- ing or pairs of training samples through mixup. By simplifying and extending previous work that operates at the token level, our method can construct virtual training samples in a broader space and achieve improved transla- tion accuracy compared to the previous state- of-the-art. In addition, we present a simple variation of the mixup strategy to better utilize the pseudo training samples created from back- translation, obtaining further improvement in performance. 1 Introduction In recent years, neural machine translation (NMT) models (Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017) have dramatically im- proved the quality of machine translation, espe- cially with the introduction of the seminal Trans- former architecture (Vaswani et al., 2017) that has become the de facto modeling choice. NMT train- ing aims to learn a parameterized function that models the prediction of the translation in a tar- get language given a source language sentence from labeled training data, which is often limited in volume especially for low-resource domains or languages. Similar to other fields in deep learn- ing, model robustness is an area of concern for NMT as a minor change in the input sentence may result in a different or incorrect translation. In practice this can happen with spelling or gram- mar errors (Provilkov et al., 2019), speech recog- nition errors (Ruiz et al., 2019; Di Gangi et al., 2019), or even a sentence of the same meaning but with a slightly different use of words or ex- pressions. Some studies (Belinkov and Bisk, 2017) have shown that the performance of NMT models can drop significantly when small perturbations are added to input sentences. This problem can be attributed to overfitting as it is difficult to reliably model the translation distribu- tion for the part of input space that has little or no training samples. There have been several attempts to address this problem by filling the space via data augmentation. One direction is to create new train- ing samples by adding perturbations at the token level (Wang et al., 2018; Belinkov and Bisk, 2017; Sperber et al., 2017; Ebrahimi et al., 2018; Li et al., 2019; Cheng et al., 2018, 2019, 2020; Levy et al., 2019), through either token insertion, deletion, and substitution operations or introducing noises to to- ken embedding vectors. Among these approaches, Cheng et al. (2019) demonstrated the effectiveness of incorporating adversarial training samples that are natural sentences, with their semantic …

Content truncated to 500 words