Token-level Adaptive Training for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1035–1046, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1035 Token-level Adaptive Training for Neural Machine Translation Shuhao Gu1,2, Jinchao Zhang3, Fandong Meng3, Yang Feng1,2∗, Wanying Xie1,4, Jie Zhou3, Dong Yu4 1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China 4 Beijing Language and Culture University, China {gushuhao19b,fengyang}@ict.ac.cn {dayerzhang,fandongmeng,withtomzhou}@tencent.com xiewanying07@gmail.com, yudong@blcu.edu.cn Abstract There exists a token imbalance phenomenon in natural language as different tokens ap- pear with different frequencies, which leads to different learning difficulties for tokens in Neural Machine Translation (NMT). The vanilla NMT model usually adopts trivial equal-weighted objectives for target tokens with different frequencies and tends to gener- ate more high-frequency tokens and less low- frequency tokens compared with the golden token distribution. However, low-frequency tokens may carry critical semantic informa- tion that will affect the translation quality once they are neglected. In this paper, we ex- plored target token-level adaptive objectives based on token frequencies to assign appropri- ate weights for each target token during train- ing. We aimed that those meaningful but rela- tively low-frequency words could be assigned with larger weights in objectives to encourage the model to pay more attention to these to- kens. Our method yields consistent improve- ments in translation quality on ZH-EN, EN- RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens where we can get 1.68, 1.02, and 0.52 BLEU increases compared with baseline, re- spectively. Further analyses show that our method can also improve the lexical diversity of translation. 1 Introduction Neural machine translation (NMT) systems (Kalch- brenner and Blunsom, 2013; Cho et al., 2014; ∗Corresponding author: Yang Feng. Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc, China. Reproducible code: https://github.com/ictnlp/TLAT-NMT. Token Order Average Reference Vanilla (Descending) Frequency NMT [0, 10%) 10, 857 81.75% 87.26% [10%, 30%) 516 11.40% 9.06% [30%, 50%) 133 3.43% 2.21% [50%, 70%) 60 1.95% 0.99% [70%, 100%] 25 1.47% 0.48% Table 1: The average frequency on the NIST training set and proportion of tokens with different frequencies in reference and the translation of the vanilla NMT model (a Transformer model) on the NIST test sets. All the target tokens (BPE sub-words with 30K merge oper- ations ) of the training set are ranked by their frequen- cies in descending order. The ’Token Order’ column represents the frequency interval ([10%, 30%) means the frequency of token is between top 10% and 30%). The ’Average Frequency’ column represents the aver- age frequencies of the tokens in each interval, which show the token imbalance phenomenon in natural lan- guage. The last two columns show the vanilla NMT model tends to generate more high-frequency tokens and less low-frequency tokens than reference. Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are data driven models, which highly depend on …

Content truncated to 500 words