Learning Confidence for Transformer-based Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2353 - 2364 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Confidence for Transformer-based Neural Machine Translation Yu Lu1,2∗, Jiali Zeng3, Jiajun Zhang1,2†, Shuangzhi Wu3 and Mu Li3 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3 Tencent Cloud Xiaowei, Beijing, China {yu.lu, jjzhang}@nlpr.ia.ac.cn {lemonzeng, frostwu, ethanlli}@tencent.com Abstract Confidence estimation aims to quantify the con- fidence of the model prediction, providing an expectation of success. A well-calibrated confi- dence estimate enables accurate failure predic- tion and proper risk measurement when given noisy samples and out-of-distribution data in real-world settings. However, this task remains a severe challenge for neural machine transla- tion (NMT), where probabilities from softmax distribution fail to describe when the model is probably mistaken. To address this problem, we propose an unsupervised confidence esti- mate learning jointly with the training of the NMT model. We explain confidence as how many hints the NMT model needs to make a correct prediction, and more hints indicate low confidence. Specifically, the NMT model is given the option to ask for hints to improve translation accuracy at the cost of some slight penalty. Then, we approximate their level of confidence by counting the number of hints the model uses. We demonstrate that our learned confidence estimate achieves high accuracy on extensive sentence/word-level quality es- timation tasks. Analytical results verify that our confidence estimate can correctly assess underlying risk in two real-world scenarios: (1) discovering noisy samples and (2) detect- ing out-of-domain data. We further propose a novel confidence-based instance-specific label smoothing approach based on our learned con- fidence estimate, which outperforms standard label smoothing1. 1 Introduction Confidence estimation has become increasingly critical with the widespread deployment of deep neural networks in practice (Amodei et al., 2016). It aims to measure the model’s confidence in the prediction, showing when it probably fails. A cali- brated confidence estimate can accurately identify ∗Work done while the author was an intern at Tencent. †Corresponding author. 1https://github.com/yulu-dada/Learned-conf-NMT Src： Ref： 爱丽莎 的 扮演者 张艳 是 国家 一 级 演员 elisa is played by zhang yan, a class-1 actress on the state level Ours：zhang yan , a figure who loves to play , is a national class actor Prob： Conf： Figure 1: An example of generated probabilities and our learned confidence estimates. The phrases in red are wrong translations. The corresponding prediction probabilities and confidence estimates are outlined in dashed boxes. The dark color indicates a large value under two evaluations. failure, further measuring the potential risk induced by noisy samples and out-of-distribution data preva- lent in real scenarios (Nguyen and O’Connor, 2015; Snoek et al., 2019). Unfortunately, neural machine translation (NMT) is reported to yield poor-calibrated confi- dence estimate (Kumar and Sarawagi, 2019; Wang et al., 2020), which is common in the application of modern neural networks (Guo et al., 2017). It implies …

Content truncated to 500 words