Normalizing Mutual Information for Robust Adaptive Training for Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8008 - 8015 December 7-11, 2022 ©2022 Association for Computational Linguistics Normalizing Mutual Information for Robust Adaptive Training for Translation Youngwon Lee1∗, Changmin Lee2, Hojin Lee2, Seung-won Hwang1† 1 Seoul National University, {ludaya,seungwonh}@snu.ac.kr 2 Kakao Enterprise, South Korea, {louis.cm,lambda.lee}@kakaoenterprise.com Abstract Despite the success of neural machine transla- tion models, tensions between fluency of opti- mizing target language modeling and source- faithfulness remain as challenges. Previ- ously, Conditional Bilingual Mutual Informa- tion (CBMI), a scoring metric for the impor- tance of target sentences and tokens, was pro- posed to encourage fluent and faithful trans- lations. The score is obtained by combining the probability from the translation model and the target language model, which is then used to assign different weights to losses from sen- tences and tokens. Meanwhile, we argue this metric is not properly normalized, for which we propose Normalized Pointwise Mutual In- formation (NPMI). NPMI utilizes an additional language model on source language to approxi- mate the joint likelihood of source-target pair and the likelihood of the source, which is then used for normalizing the score. We showed that NPMI better captures the dependence between source-target and that NPMI-based token-level adaptive training brings improvements over baselines with empirical results from En-De, De-En, and En-Ro translation tasks. 1 Introduction Neural machine translation (NMT) models have achieved remarkable performance since Vaswani et al. (2017) introduced encoder-decoder architec- ture with self- and cross-attention mechanisms. However, they were also reported to generate hal- lucinations (Lee et al., 2018; Raunak et al., 2021) in some cases, failing to balance the dual goals of improving fluency while preserving faithfulness to the source. Conditional Bilingual Mutual Information (CBMI), a metric for target tokens and sentences computed as the log quotient of the translation and the target-side language model probability, was ∗Work done during internship at Kakao Enterprise. †Corresponding author. Figure 1: An example from IWSLT14 De-En train set. While our proposed sentence-level NPMI assigns a large score (near the upper bound 1) to the faithful source- target pair and a small score (near zero, which indicates neutrality) to a rather noisy pair, sentence-level CBMI scores for the two pairs are unable to achieve that. Note that the joint log likelihood values from the two pairs are comparable, while target lengths differ a lot. proposed by Zhang et al. (2022) to guide the trans- lation model with additional signal of importance of each target token or sentence, in combination with token-level adaptive training (Gu et al., 2020) in order to achieve this goal. While CBMI score is devised to pursue this joint goal of translation, we argue that it does not take source context into account which leads to its fail- ure to provide a reliable measure of relevance for some cases. For example, Figure 1 illustrates faith- ful and noisy translations, where the former is ex- pected to have a higher score. However, sentence- level CBMI score, which is defined as aggregated token-level …

Content truncated to 500 words