Prevent the Language Model from being Overconfident in Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3456–3468 August 1–6, 2021. ©2021 Association for Computational Linguistics 3456 Prevent the Language Model from being Overconfident in Neural Machine Translation Mengqi Miao1∗, Fandong Meng2∗, Yijin Liu2, Xiao-Hua Zhou3† , and Jie Zhou2 1Peking University, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China 3Beijing International Center for Mathematical Research, National Engineering Lab for Big Data Analysis and Applications, Department of Biostatistics, Peking University, Beijing, China miaomq@pku.edu.cn, {fandongmeng, yijinliu}@tencent.com azhou@math.pku.edu.cn, withtomzhou@tencent.com Abstract The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Lan- guage Model (LM) that predicts the next to- ken only based on partial translation. De- spite its success, NMT still suffers from the hallucination problem, generating fluent but inadequate translations. The main reason is that NMT pays excessive attention to the par- tial translation while neglecting the source sentence to some extent, namely overconfi- dence of the LM. Accordingly, we define the Margin between the NMT and the LM, calculated by subtracting the predicted prob- ability of the LM from that of the NMT model for each token. The Margin is neg- atively correlated to the overconfidence de- gree of the LM. Based on the property, we propose a Margin-based Token-level Objec- tive (MTO) and a Margin-based Sentence- level Objective (MSO) to maximize the Mar- gin for preventing the LM from being over- confident. Experiments on WMT14 English- to-German, WMT19 Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improve- ments, respectively, compared to the Trans- former baseline. The human evaluation further verifies that our approaches improve transla- tion adequacy as well as fluency. 1 1 Introduction Neural Machine Translation (NMT) has achieved great success in recent years (Sutskever et al., 2014; ∗Equal contribution. This work was done when Mengqi Miao was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Corresponding author. 1Code is available at https://github.com/Mlair 77/nmt adequacy Cho et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017; Meng and Zhang, 2019; Zhang et al., 2019a; Yan et al., 2020b), which generates accurate and fluent translation through modeling the next word conditioned on both the source sentence and partial translation. However, NMT faces the hallucination problem, i.e., trans- lations are fluent but inadequate to the source sen- tences. One important reason is that the NMT model pays excessive attention to the partial trans- lation to ensure fluency while failing to translate some segments of the source sentence (Weng et al., 2020b), which is actually the overconfidence of the Language Model (LM). In the rest of this paper, the LM mentioned refers to the LM mechanism involved in NMT. Many recent studies attempt to deal with the inadequacy problem of NMT from two main as- pects. …

Content truncated to 500 words