Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2862–2872 August 1–6, 2021. ©2021 Association for Computational Linguistics 2862 Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation Yang Feng1,2 Shuhao Gu1,2 Dengji Guo1,2 Zhengxin Yang1,2 Chenze Shao1,2 ∗ 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences, Beijing, China {fengyang, gushuhao19b, guodengji19s}@ ict.ac.cn {yangzhengxin17z, shaochenze18z}@ ict.ac.cn Abstract Although teacher forcing has become the main training paradigm for neural machine transla- tion, it usually makes predictions only condi- tioned on past information, and hence lacks global planning for the future. To address this problem, we introduce another decoder, called seer decoder, into the encoder-decoder framework during training, which involves fu- ture information in target predictions. Mean- while, we force the conventional decoder to simulate the behaviors of the seer decoder via knowledge distillation. In this way, at test the conventional decoder can perform like the seer decoder without the attendance of it. Experiment results on the Chinese-English, English-German and English-Romanian trans- lation tasks show our method can outper- form competitive baselines significantly and achieves greater improvements on the bigger data sets. Besides, the experiments also prove knowledge distillation the best way to trans- fer knowledge from the seer decoder to the conventional decoder compared to adversarial learning and L2 regularization. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) has achieved great success and is draw- ing larger attention recently. Most NMT models are under the attention-based encoder-decoder frame- work which assumes there is a common seman- tic space between the source and target languages. The encoder encodes the source sentence to the common space to get its meaning, and the decoder projects the source meaning to the target space to generate corresponding target words. Whenever generating a target word at a time step, the decoder ∗The code: https://github.com/ictnlp/SeerForcingNMT needs to retrieve the attended source information and then decodes into a target word. The underline principle which makes sure the framework works is that the information hold by the source sentence and its target counterpart is equivalent. Thus the translation procedure can be considered to decom- pose source information into different pieces and then to convert each piece to a proper target word according to bilingual context. When all the infor- mation encoded in the source sentence is throughly processed, the whole translation has been gener- ated. Neural machine translation models are usually trained via maximum likelihood estimation (MLE) (Johansen and Juselius, 1990) and the operation form is known as teacher forcing (Williams and Zipser, 1989). The teacher forcing strategy per- forms one-step-ahead predictions with the past ground truth words fed as context and forces the distribution of the next prediction to approach a 0-1 distribution where the probability of the next ground …

Content truncated to 500 words