Adversarial Subword Regularization for Robust Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1945–1953 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1945 Adversarial Subword Regularization for Robust Neural Machine Translation Jungsoo Park Mujeen Sung Jinhyuk Lee† Jaewoo Kang† Korea University {jungsoo_park,mujeensung,jinhyuk_lee,kangj}@korea.ac.kr Abstract Exposing diverse subword segmentations to neural machine translation (NMT) models of- ten improves the robustness of machine trans- lation as NMT models can experience vari- ous subword candidates. However, the diver- sification of subword segmentations mostly re- lies on the pre-trained subword language mod- els from which erroneous segmentations of un- seen words are less likely to be sampled. In this paper, we present adversarial subword reg- ularization (ADVSR) to study whether gradi- ent signals during training can be a substitute criterion for exposing diverse subword seg- mentations. We experimentally show that our model-based adversarial samples effectively encourage NMT models to be less sensitive to segmentation errors and improve the perfor- mance of NMT models in low-resource and out-domain datasets. 1 Introduction Subword segmentation is a method of segment- ing an input sentence into a sequence of subword units (Sennrich et al., 2016; Wu et al., 2016; Kudo, 2018). Segmenting a word to the composition of subwords alleviates the out-of-vocabulary problem while retaining encoded sequence length compactly. Due to its effectiveness in the open vocabulary set, the method has been applied to many NLP tasks including neural machine translation (NMT) and others (Gehring et al., 2017; Vaswani et al., 2017; Devlin et al., 2019; Yang et al., 2019). Recently, Byte-Pair-Encoding(BPE) (Sennrich et al., 2016) has become one of the de facto sub- word segmentation methods. However, as BPE de- terministically segments each word into subword units, NMT models with BPE always observe the †Corresponding authors Original Text Noisy Text Input Ref Subword Segmentation of the Input Input Base SR Ours De petites fenêtre_s , une taille dés human isant_e . Subword Segmentation of the Input De petites f_en_ê_pre_s , une tail_e dés human isant_e . De petites fenêtres, une taille déshumanisante. Small windows, dehumanizing scale. De petites fenêpres, une taile déshumanisante. Small chicks, a dehumanizing carve. Small fentress, a dehumanizing tail. Small windows, a dehumanizing size. Figure 1: NMT models suffer from typos (character drop, character replacement) in the source text due to the unseen subword compositions (‘_’ denotes segmen- tation). On the other hand, Ours correctly decodes them. Base: standard training, SR: subword regular- ization (Kudo, 2018) same segmentation result for each word and often fail to learn diverse morphological features. In this regard, Kudo (2018) proposed subword regulariza- tion, a training method that exposes multiple seg- mentations using a unigram language model. Start- ing from machine translation, it has been shown that subword regularization can improve the robust- ness of NLP models in various tasks (Kim, 2019; Provilkov et al., 2019; Drexler and Glass, 2019; Müller et al., 2019). However, subword regularization relies on the unigram language models to sample candidates, where the language models are optimized based on the corpus-level statistics from training data with …

Content truncated to 500 words