Evaluating Robustness to Input Perturbations for Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8538–8544 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8538 Evaluating Robustness to Input Perturbations for Neural Machine Translation Xing Niu, Prashant Mathur, Georgiana Dinu, Yaser Al-Onaizan Amazon AI {xingniu,pramathu,gddinu,onaizan}@amazon.com Abstract Neural Machine Translation (NMT) models are sensitive to small perturbations in the in- put. Robustness to such perturbations is typ- ically measured using translation quality met- rics such as BLEU on the noisy input. This paper proposes additional metrics which mea- sure the relative degradation and changes in translation when small perturbations are added to the input. We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed. Results show that our proposed met- rics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used. 1 Introduction Recent work has pointed out the challenges in building robust neural network models (Goodfel- low et al., 2015; Papernot et al., 2016). For Neural Machine Translation (NMT) in particular, it has been shown that NMT models are brittle to small perturbations in the input, both when these pertur- bations are synthetically created or generated to mimic real data noise (Belinkov and Bisk, 2018). Consider the example in Table 1 where an NMT model generates a worse translation as a conse- quence of only one character changing in the input. Improving robustness in NMT has received a lot of attention lately with data augmentation (Sper- ber et al., 2017; Belinkov and Bisk, 2018; Vaib- hav et al., 2019; Liu et al., 2019; Karpukhin et al., 2019) and adversarial training methods (Cheng et al., 2018; Ebrahimi et al., 2018; Cheng et al., 2019; Michel et al., 2019) as some of the more popular approaches used to increase robustness in neural network models. In this paper, we focus on one class of meth- ods, subword regularization, which addresses NMT Original input Se kyll¨a tuntuu sangen luultavalta. Translation It certainly seems very likely. Perturbed input Se kyll¨a tumtuu sangen luultavalta. Translation It will probably darken quite probably. Reference It certainly seems probable. Table 1: An example of NMT English translations for a Finnish input and its one-letter misspelled version. robustness without introducing any changes to the architectures or to the training regime, solely through dynamic segmentation of input into sub- words (Kudo, 2018; Provilkov et al., 2019). We provide a comprehensive comparison of these meth- ods on several language pairs and under different noise conditions on robustness-focused metrics. Previous work has used translation quality mea- sures such as BLEU on noisy input as an indicator of robustness. Absolute model performance on noisy input is important, and we believe this is an appropriate measure for noisy domain evalua- tion (Michel and Neubig, 2018; Berard et al., 2019; Li et al., 2019). However, it does not disentangle model quality from the relative degradation under added noise. For this reason, we propose two additional mea- …

Content truncated to 500 words