Disentangling Uncertainty in Machine Translation Evaluation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8622 - 8641 December 7-11, 2022 ©2022 Association for Computational Linguistics Disentangling Uncertainty in Machine Translation Evaluation Chrysoula Zerva1,4 Taisiya Glushkova1,4 Ricardo Rei2,3,4 André F. T. Martins1,2,4 1Instituto de Telecomunicações 2Unbabel 3INESC-ID 4Instituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit) {chrysoula.zerva, taisiya.glushkova, ricardo.rei, andre.t.martins}@tecnico.ulisboa.pt Abstract Trainable evaluation metrics for machine trans- lation (MT) exhibit strong correlation with hu- man judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. Recent work has attempted to mitigate this with simple uncertainty quantification techniques (Monte Carlo dropout and deep ensembles), however these techniques (as we show) are limited in several ways – for example, they are unable to distinguish between different kinds of un- certainty, and they are time and memory con- suming. In this paper, we propose more power- ful and efficient uncertainty predictors for MT evaluation, and we assess their ability to target different sources of aleatoric and epistemic un- certainty. To this end, we develop and compare training objectives for the COMET metric to enhance it with an uncertainty prediction out- put, including heteroscedastic regression, di- vergence minimization, and direct uncertainty prediction. Our experiments show improved results on uncertainty prediction for the WMT metrics task datasets, with a substantial reduc- tion in computational costs. Moreover, they demonstrate the ability of these predictors to address specific uncertainty causes in MT eval- uation, such as low quality references and out- of-domain data.1 1 Introduction Trainable neural-based metrics, such as COMET or BLEURT (Rei et al., 2020a; Sellam et al., 2020a), hold great promise for MT evaluation (Freitag et al., 2021b). For system comparison, they sur- pass or complement traditional lexical metrics such as BLEU (Papineni et al., 2002), and at a segment level, they show higher correlations with human judgments, with and without access to references (Kepler et al., 2019; Thompson and Post, 2020; Ranasinghe et al., 2020). 1Our code and data is available at: https://github.com/ deep-spin/uncertainties_MT_eval 0.00 0.25 0.50 0.75 1.00 1.25 DUP HTS MCD DE News (in-domain) TEDtalks (out-of-domain) Figure 1: Epistemic uncertainty caused by out-of- domain data. We show sharpness (average uncertainty) on two English-Russian test sets from the WMT21 met- rics task: an in-domain dataset (News) and an out-of- domain dataset (TED talks). Our proposed method that handles epistemic uncertainty (Direct Uncertainty Pre- dictor – DUP) exhibits higher uncertainty on the out-of- domain dataset, as expected. The heteroscedastic (HTS) predictor, which detects aleatoric, but not epistemic un- certainty, has similar uncertainty in both datasets, and the MC dropout (MCD) and deep ensemble (DE) base- lines, surprisingly, has the opposite behavior. However, trainable MT evaluation metrics are not always trustworthy: For example, they can be unreliable in out-of-domain data and low re- source languages, and sometimes they disregard specific error types, attributing high scores to low quality translations (Amrhein and Sennrich, 2022). Hence, we need a measure of confidence over their quality predictions for each segment, so that …

Content truncated to 500 words