Towards Robust Neural Machine Translation with Iterative Scheduled Data-Switch Training

Proceedings of the 29th International Conference on Computational Linguistics, pages 5266–5277 October 12–17, 2022. 5266 Towards Robust Neural Machine Translation with Iterative Scheduled Data-Switch Training Zhongjian Miao1 †∗ Xiang Li2† Liyan Kang1 Wen Zhang2 Chulun Zhou1 Yidong Chen1‡ Bin Wang2 Min Zhang3 Jinsong Su1,4‡ 1School of Informatics, Xiamen University, China 2Xiaomi AI Lab, China 3Harbin Institute of Technology, Shenzhen, China 4Pengcheng Lab, China miaozhongjian@stu.xmu.edu.cn lixiang21@xiaomi.com {ydchen,jssu}@xmu.edu.cn Abstract Most existing methods on robust neural ma- chine translation (NMT) construct adversarial examples by injecting noise into authentic ex- amples and indiscriminately exploit two types of examples. They require the model to trans- late both the authentic source sentence and its adversarial counterpart into the identical tar- get sentence within the same training stage, which may be a suboptimal choice to achieve robust NMT. In this paper, we first conduct a preliminary study to confirm this claim and further propose an Iterative Scheduled Data- switch Training Framework to mitigate this problem. Specifically, we introduce two train- ing stages, iteratively switching between au- thentic and adversarial examples. Compared with previous studies, our model focuses more on just one type of examples at each single stage, which can better exploit authentic and adversarial examples, and thus obtaining a bet- ter robust NMT model. Moreover, we intro- duce an improved curriculum learning method with a sampling strategy to better schedule the process of noise injection. Experimental results show that our model significantly sur- passes several competitive baselines on four translation benchmarks. Our source code is available at https://github.com/ DeepLearnXMU/RobustNMT-ISDST. 1 Introduction In recent years, neural machine translation (NMT) has achieved great success (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). Usu- ally, the NMT models are trained on clean parallel corpus and thus achieve promising performance under clean inputs. However, small perturbations, such as replacing words in the input sentences, can mislead the trained model to generate incorrect ∗Work was done when interning at Xiaomi AI Lab. †Equal Contribution. ‡Corresponding Author. translations (Belinkov and Bisk, 2018). In real- world scenarios, it is often required to deal with such sentences. Thus, it has important academic value and application prospects to design a robust NMT model for both clean and noisy inputs. To reach this goal, some researchers explore data-oriented approaches focusing on construct- ing adversarial examples (Cheng et al., 2020; Zou et al., 2020). Generally, adversarial examples are used to augment the authentic dataset or fine-tune an NMT model pre-trained on the authentic dataset to improve robustness. Although data-oriented ap- proaches are simple and efficient, they leverage ad- versarial examples coarsely, as concluded by Wang et al. (2021a) and Passban et al. (2021), which can not reach the full potential of these examples. Besides, researchers also study model-oriented approaches. Some design additional model com- ponents to correct noisy inputs (Zhou et al., 2019; Qin et al., 2021; Wang et al., 2021a). There are more studies exploring training strategies for ro- bust NMT, including multi-task learning (Zhou et al., 2019; Zhang et al., 2020), …

Content truncated to 500 words