Searching for COMETINHO: The Little Metric That Could

Searching for COMETINHO: The Little Metric That Could Ricardo Rei∗,1,2,3 Ana C Farinha∗,1 Jos´e G. C. de Souza∗,1 Pedro G. Ramos2 Andr´e F. T. Martins1,3,4 Luisa Coheur2,3 Alon Lavie1 1Unbabel 2INESC-ID 3Instituto Superior T´ecnico 4Instituto de Telecomunicac¸˜oes {ricardo.rei, catarina.farinha, jose.souza}@unbabel.com Abstract Recently proposed neural-based machine translation evaluation metrics, such as COMET and BLEURT, exhibit much higher correlations with human judgments than traditional lexical overlap metrics. How- ever, they require large models and are computationally very costly, preventing their application in scenarios where one has to score thousands of translation hy- potheses (e.g. outputs of multiple sys- tems or different hypotheses of the same system, as in minimum Bayes risk decod- ing). In this paper, we introduce several techniques, based on pruning and knowl- edge distillation, to create more compact and faster COMET versions—which we dub COMETINHO. First, we show that just by optimizing the code through the use of caching and length batching we can reduce inference time between 39% and 65% when scoring multiple systems. Second, we show that pruning COMET can lead to a 21% model reduction with- out affecting the model’s accuracy be- yond 0.015 Kendall τ correlation. Finally, we present DISTIL-COMET, a lightweight distilled version that is 80% smaller and 2.128x faster while attaining a perfor- mance close to the original model. Our code is available at: https://github. com/Unbabel/COMET © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. * Corresponding authors. 1 Introduction Traditional metrics for machine translation (MT) evaluation rely on lexical similarity between a given hypothesis and a reference translation. Met- rics such as BLEU (Papineni et al., 2002) and CHRF (Popovi´c, 2015) remain popular due to ef- ficient memory usage and fast computational per- formance, even though several studies have shown that they correlate poorly with human judgements, specially for high quality MT (Ma et al., 2019; Mathur et al., 2020a). In contrast, neural fine-tuned metrics on top of pre-trained models such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) (e.g BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020) have demonstrated significant improve- ments in comparison to other metrics (Mathur et al., 2020b; Kocmi et al., 2021; Freitag et al., 2021b). The improvements made them good can- didates for revisiting promising research directions where the metric plays a more central role in can- didate selection during decoding, such as N-best reranking (Ng et al., 2019; Bhattacharyya et al., 2021; Fernandes et al., 2022) and minimum Bayes risk (MBR) decoding (Eikema and Aziz, 2021; M¨uller and Sennrich, 2021). Nonetheless, the complexity of such strategies using metrics based on large transformer models can become impracti- cal for a large set of MT hypotheses. In this paper, we describe several experiments that attempt to reduce COMET computational cost and model size to make it more efficient at in- ference. Our techniques are particularly useful in settings where we have multiple translations from different systems on the same source …

Content truncated to 500 words