Effective Adversarial Regularization for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 204–210 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 204 Effective Adversarial Regularization for Neural Machine Translation Motoki Sato1, Jun Suzuki2,3, Shun Kiyono3,2 1Preferred Networks, Inc., 2Tohoku University, 3RIKEN Center for Advanced Intelligence Project sato@preferred.jp, jun.suzuki@ecei.tohoku.ac.jp, shun.kiyono@riken.jp Abstract A regularization technique based on adversar- ial perturbation, which was initially developed in the field of image processing, has been suc- cessfully applied to text classification tasks and has yielded attractive improvements. We aim to further leverage this promising method- ology into more sophisticated and critical neu- ral models in the natural language processing field, i.e., neural machine translation (NMT) models. However, it is not trivial to apply this methodology to such models. Thus, this paper investigates the effectiveness of several pos- sible configurations of applying the adversar- ial perturbation and reveals that the adversar- ial regularization technique can significantly and consistently improve the performance of widely used NMT models, such as LSTM- based and Transformer-based models.1 1 Introduction The existence of (small) perturbations that in- duce a critical prediction error in machine learn- ing models was first discovered and discussed in the field of image processing (Szegedy et al., 2014). Such perturbed inputs are often referred to as adversarial examples in the literature. Sub- sequently, Goodfellow et al. (2015) proposed a learning framework that simultaneously leverages adversarial examples as additional training data for reducing the prediction errors. This learning framework is referred to as adversarial training. In the field of natural language processing (NLP), the input is a sequence of discrete symbols, such as words or sentences. Since it is unreason- able to add a small perturbation to the symbols, ap- plying the idea of adversarial training to NLP tasks has been recognized as a challenging problem. Re- cently, Miyato et al. (2017) overcame this problem 1Our code for replicating the experiments in this paper is available at the following URL: https://github.com/ pfnet-research/vat_nmt Encoder Decoder ��� ��� !" #$" !% #$% !& #$& '( #$( ) '" #$" ) '* #$+ ) ��� ," ,% ,+-" Figure 1: An intuitive sketch that explains how we add adversarial perturbations to a typical NMT model structure for adversarial regularization. The definitions of ei and fj can be found in Eq. 2. Moreover, those of ˆri and ˆr0 j are in Eq. 8 and 13, respectively. and reported excellent performance improvements on multiple benchmark datasets of text classifica- tion task. The key idea of their success is to apply adversarial perturbations into the input embedding layer instead of the inputs themselves as used in image processing tasks. An important implication of their study is that their method can be inter- preted as a regularization method, and thus, they do not focus on generating adversarial examples. We refer to this regularization technique as adver- sarial regularization. We aim to further leverage this promising methodology into more sophisticated and criti- cal neural models, i.e., neural machine transla- tion …

Content truncated to 500 words