Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2878 - 2889 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Confidence Based Bidirectional Global Context Aware Training Framework for Neural Machine Translation Chulun Zhou1†, Fandong Meng2, Jie Zhou2, Min Zhang3, Hongji Wang1, Jinsong Su1,4∗ 1School of Informatics, Xiamen University, Xiamen 2Pattern Recognition Center, WeChat AI, Tencent Inc, China 3Harbin Institute of Technology, Shenzhen 4Pengcheng Lab, Shenzhen clzhou@stu.xmu.edu.cn {fandongmeng,withtomzhou}@tencent.com zhangminmt@hotmail.com {whj,jssu}@xmu.edu.cn Abstract Most dominant neural machine translation (NMT) models are restricted to make predic- tions only according to the local context of preceding words in a left-to-right manner. Al- though many previous studies try to incor- porate global information into NMT models, there still exist limitations on how to effec- tively exploit bidirectional global context. In this paper, we propose a Confidence Based Bidirectional Global Context Aware (CB- BGCA) training framework for NMT, where the NMT model is jointly trained with an auxiliary conditional masked language model (CMLM). The training consists of two stages: (1) multi-task joint training; (2) confidence based knowledge distillation. At the first stage, by sharing encoder parameters, the NMT model is additionally supervised by the signal from the CMLM decoder that con- tains bidirectional global contexts. More- over, at the second stage, using the CMLM as teacher, we further pertinently incorpo- rate bidirectional global context to the NMT model on its unconfidently-predicted target words via knowledge distillation. Experi- mental results show that our proposed CB- BGCA training framework significantly im- proves the NMT model by +1.02, +1.30 and +0.57 BLEU scores on three large-scale translation datasets, namely WMT’14 English- to-German, WMT’19 Chinese-to-English and WMT’14 English-to-French, respectively. 1 Introduction In recent years, Neural Machine Translation (NMT) has achieved great progress and attracted more attention. Most dominant NMT models mainly adopt an encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017; Meng and Zhang, 2019; Song et al., 2019; †This work is done when Chulun Zhou was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. *Corresponding author Miao et al., 2021) with the teacher-forcing strat- egy (Goodfellow et al., 2016) for training. Despite its success, the unidirectional property of teacher- forcing strategy restricts NMT models to only focus on the local context, i.e., the preceding words of the to-be-predicted target word at each decoder step. Apparently, this strategy tends to be limited be- cause word dependencies are always bidirectional involving both preceding and succeeding words on the target side. To address this issue, many previous researches attempt to exploit global information on the target side (Liu et al., 2016; Zhang et al., 2016; Serdyuk et al., 2018; Zhang et al., 2018; Su et al., 2018; Zhang et al., 2019a,b; Su et al., 2019; Zhou et al., 2019; Zhang et al., 2020). Typically, they intro- duce the modelling of target-side global context in the reverse direction by pairing the conventional left-to-right (L2R) NMT model with a right-to-left (R2L) auxiliary model. However, in these methods, the …

Content truncated to 500 words