Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1125–1141 November 20–23, 2022. ©2022 Association for Computational Linguistics 1125 Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET Chantal Amrhein1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh {amrhein,sennrich}@cl.uzh.ch Abstract Neural metrics have achieved impressive corre- lation with human judgements in the evaluation of machine translation systems, but before we can safely optimise towards such metrics, we should be aware of (and ideally eliminate) bi- ases toward bad translations that receive high scores. Our experiments show that sample- based Minimum Bayes Risk decoding can be used to explore and quantify such weaknesses. When applying this strategy to COMET for en→de and de→en, we find that COMET mod- els are not sensitive enough to discrepancies in numbers and named entities. We further show that these biases are hard to fully remove by simply training on additional synthetic data and release our code and data for facilitating further experiments.1 1 Introduction Recently, neural machine translation evaluation metrics have reached better correlation scores with human evaluators than surface-level metrics like BLEU (Papineni et al., 2002). In particular, COMET (Rei et al., 2020a) has shown significant potential as a leading evaluation metric both in shared tasks (Mathur et al., 2020; Freitag et al., 2021b) and other studies on machine translation evaluation metrics (Kocmi et al., 2021). The main benefits of such neural metrics are that they do not rely on surface-level similarity to a reference translation and that some of them operate in a mul- tilingual representation space. This also allows for comparing translations to the source sentence. A recent evaluation as part of the WMT 2021 metrics shared task (Freitag et al., 2021b) suggests that neural metrics are also less susceptible to many weaknesses of earlier non-neural metrics, e.g. an antonym in the translation hurting the BLEU score exactly the same amount as a synonym. However, 1https://github.com/ZurichNLP/ mbr-sensitivity it is still unclear whether or not these metrics also introduce new biases that are harder to detect since they are essentially “black box” metrics that do not explain why a certain score is attributed to a trans- lation. Failing to identify these biases in neural metrics could lead the community to optimise to- wards metric “blind spots”, either directly through reward-based training methods such as Minimum Risk Training (Shen et al., 2016), or more slowly by basing modelling choices on metric scores. It is therefore worthwhile to find new means to uncover weaknesses of neural machine translation metrics. In this paper, we show that sampling-based Min- imum Bayes Risk (MBR) decoding - where a pool of samples are compared against each other using a machine translation evaluation metric as a utility function - can render blind spots of these metrics more observable. When applying COMET as the utility …

Content truncated to 500 words