Reducing Length Bias in Scoring Neural Machine Translation via a Causal Inference Method

Reducing Length Bias in Scoring Neural Machine Translation via a Causal Inference Method Xuewen Shi1,2, Heyan Huang1,2, Ping Jian1,2∗, Yi-Kun Tang1,2 1School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China 2Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications {xwshi, hhy63, pjian, tangyk}@bit.edu.cn Abstract Neural machine translation (NMT) usually employs beam search to expand the searching space and obtain more translation candidates. However, the increase of the beam size often suffers from plenty of short translations, resulting in dramatical decrease in translation quality. In this paper, we handle the length bias problem through a perspective of causal inference. Specifically, we regard the model generated translation score S as a degraded true translation quality affected by some noise, and one of the confounders is the translation length. We apply a Half-Sibling Re- gression method to remove the length effect on S, and then we can obtain a debiased translation score without length information. The proposed method is model agnostic and unsupervised, which is adaptive to any NMT model and test dataset. We conduct the experiments on three translation tasks with different scales of datasets. Experimental results and further analyses show that our approaches gain comparable performance with the empirical baseline methods. 1 Introduction Recently, with the renaissance of deep learning, end-to-end neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015) has gained remarkable performances (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017). NMT models are usually built upon an encoder-decoder framework (Cho et al., 2014): the encoder reads an input sequence x = {x1, ..., xTx} into a hidden memory H, and the decoder is designed to model a probability over the translation y = {y1, ..., yTˆy} by: P(y|x) = Ty � t=1 P(yt|y<t, H). (1) Most existing NMT approaches employ beam search to obtain more translation candidates and then gain a better translation hypothesis ˆy = {ˆy1, · · · , ˆyTˆy} by ranking the translation candidates set ˆY = {ˆy1, · · · , ˆyb} across a score function s(ˆy): s(ˆy) = Tˆy � t=1 log P(ˆyt|x; θ), (2) where b is the beam size and θ is the parameter set of the NMT model. However, continuously increasing the beam size has been shown to degrade performances and lead to short translations (Koehn and Knowles, 2017). One decisive reason is that the large search space is easy to introduce more short ˆy, and the shorter ˆy tends to be scored higher under s(ˆy) in Eq. (2). Pre- vious efforts usually deal with the above length bias problem by two mechanisms: i) performing length normalization on s(ˆy) via dividing s(ˆy) by the length penalty lp, i.e. s′(ˆy) ← s(ˆy)/lp (Boulanger- Lewandowski et al., 2013; Jean et al., 2015; Koehn and Knowles, 2017; Yang et al., 2018; Meister et al., 2020), and ii) adding an additional length-related reward r to s(ˆy), i.e. s′(ˆy) ← s(ˆy) + γ · r (Li and Jurafsky, 2016; He et …

Content truncated to 500 words