Prediction Difference Regularization against Perturbation for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7665 - 7675 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Prediction Difference Regularization against Perturbation for Neural Machine Translation Dengji Guo12, Zhengrui Ma12, Min Zhang3, Yang Feng12∗, 1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China 3 Harbin Institute of Technology, Shenzhen, China {guodengji19s,mazhengrui21b,fengyang}@ict.ac.cn zhangmin2021@hit.edu.cn Abstract Regularization methods applying input pertur- bation have drawn considerable attention and have been frequently explored for NMT tasks in recent years. Despite their simplicity and ef- fectiveness, we argue that these methods are limited by the under-fitting of training data. In this paper, we utilize prediction difference for ground-truth tokens to analyze the fitting of token-level samples and find that under- fitting is almost as common as over-fitting. We introduce prediction difference regularization (PD-R), a simple and effective method that can reduce over-fitting and under-fitting at the same time. For all token-level samples, PD- R minimizes the prediction difference between the original pass and the input-perturbed pass, making the model less sensitive to small in- put changes, thus more robust to both pertur- bations and under-fitted training data. Exper- iments on three widely used WMT transla- tion tasks show that our approach can signifi- cantly improve over existing perturbation reg- ularization methods. On WMT16 En-De task, our model achieves 1.80 SacreBLEU improve- ment over vanilla transformer. 1 Introduction Neural machine translation models have achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Despite their efficiency and superb performance, NMT models are prone to over-fitting that universal regularization techniques such as dropout (Hinton et al., 2012) and label smoothing (Szegedy et al., 2016) have been indis- pensable. However, over-fitting is still a signifi- cant problem for NMT, especially for small and medium tasks, which motivates researchers to con- stantly explore more specialized and sophisticated regularization techniques. ∗Yang Feng is the corresponding author of the paper. Particularly, regularization methods applying in- put perturbation have been frequently explored for NMT models in recent years (Bengio et al., 2015; Wu et al., 2019; Sato et al., 2019; Takase and Kiy- ono, 2021). In these methods, neural models are trained to maximize the likelihood of perturbed samples that perturbed by a certain type of per- turbations, with a primary intention to enhance model’s robustness to perturbations, since neural models have been discovered fragile to small input noises (Szegedy et al., 2014; Liang et al., 2018; Belinkov and Bisk, 2018). In the past few years, many types of perturbations have been proposed to machine translation and been shown effective, in- cluding word-dropout (Gal and Ghahramani, 2016), word-replacement (Bengio et al., 2015; Wu et al., 2019) and adversarial perturbation (Miyato et al., 2017; Sato et al., 2019), etc. In this paper, unlike previous works which are devoted to finding stronger perturbations and more appropriate …

Content truncated to 500 words