Robust Neural Machine Translation with Doubly Adversarial Inputs

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324–4333 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4324 Robust Neural Machine Translation with Doubly Adversarial Inputs Yong Cheng, Lu Jiang, and Wolfgang Macherey Google AI {chengyong, lujiang, wmach}@google.com Abstract Neural machine translation (NMT) often suf- fers from the vulnerability to noisy perturba- tions in the input. We propose an approach to improving the robustness of NMT mod- els, which consists of two parts: (1) attack the translation model with adversarial source examples; (2) defend the translation model with adversarial target inputs to improve its robustness against the adversarial source in- puts. For the generation of adversarial inputs, we propose a gradient-based method to craft adversarial examples informed by the transla- tion loss over the clean inputs. Experimen- tal results on Chinese-English and English- German translation tasks demonstrate that our approach achieves significant improvements (2.8 and 1.6 BLEU points) over Transformer on standard clean benchmarks as well as ex- hibiting higher robustness on noisy data. 1 Introduction In recent years, neural machine translation (NMT) has achieved tremendous success in advancing the quality of machine translation (Wu et al., 2016; Hieber et al., 2017). As an end-to-end sequence learning framework, NMT consists of two impor- tant components, the encoder and decoder, which are usually built on similar neural networks of dif- ferent types, such as recurrent neural networks (Sutskever et al., 2014; Bahdanau et al., 2015; Chen et al., 2018), convolutional neural networks (Gehring et al., 2017), and more recently on trans- former networks (Vaswani et al., 2017). To over- come the bottleneck of encoding the entire input sentence into a single vector, an attention mech- anism was introduced, which further enhanced translation performance (Bahdanau et al., 2015). Deeper neural networks with increased model ca- pacities in NMT have also been explored and shown promising results (Bapna et al., 2018). Input 他(她)一个残疾人，我女儿身体 好好地。 Original he is a handicapped person, my Output daughter is in good health. ✓ Perturbed one of her handicapped people, Output my daughter is in good health. × Table 1: An example of Transformer NMT translation result for an input and its perturbed input by replacing “他(he)” to “她(she)”. Despite these successes, NMT models are still vulnerable to perturbations in the input sentences. For example, Belinkov and Bisk (2018) found that NMT models can be immensely brittle to small perturbations applied to the inputs. Even if these perturbations are not strong enough to alter the meaning of an input sentence, they can neverthe- less result in different and often incorrect transla- tions. Consider the example in Table 1, the Trans- former model will generate a worse translation (re- vealing gender bias) for a minor change in the in- put from “he” to “she”. Perturbations originate from two sources: (a) natural noise in the anno- tation and (b) artificial deviations generated by at- tack models. In this paper, we do not distinguish the source of a perturbation and term perturbed …

Content truncated to 500 words