MS-COMET: More and Better Human Judgements Improve Metric Performance

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 541 - 548 December 7-8, 2022 ©2022 Association for Computational Linguistics MS-COMET: More and Better Human Judgements Improve Metric Performance Tom Kocmi Hitokazu Matsushita Christian Federmann Microsoft, 1 Microsoft Way, Redmond, WA 98052, USA {tomkocmi,himatsus,chrife}@microsoft.com Abstract We develop two new metrics that build on top of the COMET architecture. The main contri- bution is collecting a ten-times larger corpus of human judgements than COMET and inves- tigating how to filter out problematic human judgements. We propose filtering human judge- ments where human reference is statistically worse than machine translation. Furthermore, we average scores of all equal segments evalu- ated multiple times. The results comparing au- tomatic metrics on source-based DA and MQM- style human judgement show state-of-the-art performance on a system-level pair-wise sys- tem ranking. We release both of our metrics for public use.1 1 Introduction Automatic metrics for machine translation (MT) evaluation are commonly used as the primary tool for comparing the translation quality of MT sys- tems, often without evaluating systems with the human judgement that can be expensive and time- consuming (Marie et al., 2021). Therefore, study- ing and developing metrics that correlate well with human judgement is critical. There is an increasing effort in the evaluation of automatic MT metrics, leading with the annual evaluation of metrics at the WMT conference (Fre- itag et al., 2021b,a; Kocmi et al., 2021; Mathur et al., 2020b). Most research has focused on com- paring segment-level or system-level correlations between absolute metric scores and human judge- ments. However, Mathur et al. (2020a) emphasize that this scenario is not identical to the everyday use of metrics, where instead, researchers and prac- titioners use automatic scores to compare pairs of systems. For example, when claiming a new state- of-the-art, evaluating different model architectures, 1https://github.com/MicrosoftTranslator/ MS-Comet and deciding whether to publish results or deploy new production systems. In this work, we focus on training automatic metric based on COMET architecture (Rei et al., 2020) utilizing a large internal trainset of human segment-level judgements. Additionally, we evalu- ate the metrics in a pair-wise system-level evalua- tion against human judgement. We develop two metrics: MS-COMET intended for reference-based evaluating systems, while MS- COMET-QE is designed for quality estimation or source-based evaluation. We use the suffix "-22" to differentiate the models from potential future releases. 2 Related work There are two main categories of automatic MT metrics: (1) string-based metrics and (2) metrics using pretrained models. The former compares the coverage of various substrings between the human- generated reference and MT translations, this group includes metrics such as ChrF (Popovi´c, 2015), BLEU (Papineni et al., 2002), or TER (Snover et al., 2006). String-based methods largely depend on the quality of reference translations. However, their advantage is that their performance is predictable as it can easily diagnose which substrings affect the score the most. The latter category of pretrained methods con- sists of metrics that usually use pretrained models to evaluate the quality of MT translations given …

Content truncated to 500 words