Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2377 - 2389 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation Songming Zhang1∗, Yijin Liu2∗, Fandong Meng2, Yufeng Chen1†, Jinan Xu1, Jian Liu1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {zhangsongming,chenyf,jaxu,jianliu}@bjtu.edu.cn, {yijinliu,fandongmeng,withtomzhou}@tencent.com Abstract Token-level adaptive training approaches can alleviate the token imbalance problem and thus improve neural machine translation, through re-weighting the losses of different target tokens based on specific statistical metrics (e.g., token frequency or mutual information). Given that standard translation models make predictions on the condition of previous tar- get contexts, we argue that the above statisti- cal metrics ignore target context information and may assign inappropriate weights to tar- get tokens. While one possible solution is to directly take target contexts into these statis- tical metrics, the target-context-aware statis- tical computing is extremely expensive, and the corresponding storage overhead is unreal- istic. To solve the above issues, we propose a target-context-aware metric, named condi- tional bilingual mutual information (CBMI), which makes it feasible to supplement target context information for statistical metrics. Par- ticularly, our CBMI can be formalized as the log quotient of the translation model proba- bility and language model probability by de- composing the conditional joint distribution. Thus CBMI can be efficiently calculated dur- ing model training without any pre-specific statistical calculations and large storage over- head. Furthermore, we propose an effective adaptive training approach based on both the token- and sentence-level CBMI. Experimen- tal results on WMT14 English-German and WMT19 Chinese-English tasks show our ap- proach can significantly outperform the Trans- former baseline and other related methods. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017; Meng and Zhang, 2019; Liu et al., 2021a,b) ∗ Equal contribution. Work was done when Songming were interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Yufeng Chen is the corresponding author. Source: Target: píng wěn jiā sù huí fù zhèng cháng xíng shǐ sù dù , róng rù nǐ zhōu wéi de chē liú 。 Return to normal traffic speed by accelerating smoothly to blend with the traffic around you . 4.844 3.670 23262 Frequency BMI CBMI 1.508 1.508 23262 平稳 加速 回复 正常 行驶 速度 , 融入 你 周围 的 车流 。 Pinyin: Figure 1: An example from the WMT19 Zh-En train- ing set. Despite the different mappings from the source sentence, existing target-context-free metrics (i.e., fre- quency and BMI) equally assess the two ‘traffic’ tokens, while our CBMI can distinguish the different dependen- cies of the two tokens on the source sentence with the guidance of target contexts. has made remarkable achievements in recent years. Generally, NMT models are trained to maximize the likelihood of the next target token given ground- truth tokens as inputs (Johansen and Juselius, 1990; …

Content truncated to 500 words