Scheduled Sampling Based on Decoding Steps for Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3285–3296 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3285 Scheduled Sampling Based on Decoding Steps for Neural Machine Translation Yijin Liu12, Fandong Meng2, Yufeng Chen1 Jinan Xu1 and Jie Zhou2 1Beijing Jiaotong University, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {yijinliu, fandongmeng, withtomzhou}@tencent.com {jaxu,chenyf}@bjtu.edu.cn Abstract Scheduled sampling is widely used to mitigate the exposure bias problem for neural machine translation. Its core motivation is to simulate the inference scene during training by replac- ing ground-truth tokens with predicted tokens, thus bridging the gap between training and in- ference. However, vanilla scheduled sampling is merely based on training steps and equally treats all decoding steps. Namely, it simulates an inference scene with uniform error rates, which disobeys the real inference scene, where larger decoding steps usually have higher error rates due to error accumulations. To alleviate the above discrepancy, we propose scheduled sampling methods based on decoding steps, in- creasing the selection chance of predicted to- kens with the growth of decoding steps. Con- sequently, we can more realistically simulate the inference scene during training, thus bet- ter bridging the gap between training and in- ference. Moreover, we investigate scheduled sampling based on both training steps and de- coding steps for further improvements. Ex- perimentally, our approaches significantly out- perform the Transformer baseline and vanilla scheduled sampling on three large-scale WMT tasks. Additionally, our approaches also gen- eralize well to the text summarization task on two popular benchmarks. 1 Introduction Neural Machine Translation (NMT) has made promising progress in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). Generally, NMT models are trained to maximize 1To calculate the precision for training, we strictly match predicted tokens with ground-truth tokens word by word. When inference, we relax the strict matching to the fuzzy matching within a local window of size 3, and truncate or pad hypotheses to the same length of golden references. We also explore n-gram matching in preliminary experiments and observe analogical results with different n. For simplicity, we use the above unigram matching to calculate the translation precision (similarly for the error rate) in all experiments. Figure 1: The translation precision for training (blue line) and inference (red line) at each decoding step. The gap between training and inference (black line) in- creases rapidly with the growth of decoding steps. We randomly sample 100k training data from WMT 2014 EN-DE and report the average precision of 1k tokens for each decoding step1. the likelihood of next token given previous golden tokens as inputs, i.e., teacher forcing (Salakhut- dinov, 2014). However, at the inference stage, golden tokens are unavailable. The model is ex- posed to an unseen data distribution generated by itself. This discrepancy between training and infer- ence is named as the exposure bias problem (Ran- zato et al., 2016). With the growth of decoding steps, such discrepancy becomes more problem- atic due to error accumulations (Zhou et …

Content truncated to 500 words