On the Inference Calibration of Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3070–3079 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3070 On the Inference Calibration of Neural Machine Translation Shuo Wang⋆∗ Zhaopeng Tu⊤ Shuming Shi⊤ Yang Liu⋆† ⋆Institute for Artificial Intelligence Department of Computer Science and Technology, Tsinghua University Beijing National Research Center for Information Science and Technology ⊤Tencent AI Lab †Beijing Academy of Artificial Intelligence Beijing Advanced Innovation Center for Language Resources ⋆{wangshuo.thu, liuyang.china}@gmail.com ⊤{zptu, shumingshi}@tencent.com Abstract Confidence calibration, which aims to make model predictions equal to the true correct- ness measures, is important for neural machine translation (NMT) because it is able to offer useful indicators of translation errors in the generated output. While prior studies have shown that NMT models trained with label smoothing are well-calibrated on the ground- truth training data, we find that miscalibration still remains a severe challenge for NMT dur- ing inference due to the discrepancy between training and inference. By carefully design- ing experiments on three language pairs, our work provides in-depth analyses of the correla- tion between calibration and translation perfor- mance as well as linguistic properties of mis- calibration and reports a number of interest- ing findings that might help humans better an- alyze, understand and improve NMT models. Based on these observations, we further pro- pose a new graduated label smoothing method that can improve both inference calibration and translation performance. 1 1 Introduction Calibration requires that the probability a model assigns to a prediction (i.e., confidence) equals to the correctness measure of the prediction (i.e., ac- curacy). Calibrated models are important in user- facing applications such as natural language pro- cessing (Nguyen and O’Connor, 2015) and speech recognition (Yu et al., 2011), in which one needs to assess the confidence of a prediction. For example, in computer-assisted translation, a calibrated ma- chine translation model is able to tell a user when the model’s predictions are likely to be incorrect, which is helpful for the user to correct errors. ∗Work was done when Shuo Wang was interning at Ten- cent AI Lab under the Rhino-Bird Elite Training Program. 1The source code is available at https://github. com/shuo-git/InfECE. EnDe Dev (a) Training EnDe Dev (b) Inference Figure 1: Reliability diagrams in training and inference for the WMT14 En-De task. “Gap” denotes the differ- ence between confidence and accuracy. Smaller gaps denotes better calibrated outputs. We find that the av- erage gaps between confidence and accuracy are much larger in inference than in training (i.e., 15.83 > 1.39). The study of calibration on classification tasks has a long history, from statistical machine learn- ing (Platt et al., 1999; Niculescu-Mizil and Caru- ana, 2005) to deep learning (Guo et al., 2017). However, calibration on structured generation tasks such as neural machine translation (NMT) has not been well studied. Recently, M¨uller et al. (2019) and Kumar and Sarawagi (2019) studied the cali- bration of NMT in the training setting, and found that NMT trained with label smoothing (Szegedy et al., …

Content truncated to 500 words