Assessing Reference-Free Peer Evaluation for Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1158–1171 June 6–11, 2021. ©2021 Association for Computational Linguistics 1158 Assessing Reference-Free Peer Evaluation for Machine Translation Sweta Agrawal1, George Foster2, Markus Freitag2, Colin Cherry2 1Department of Computer Science, University of Maryland 2Google Research sweagraw@umd.edu, {fosterg, freitag, colincherry}@google.com Abstract Reference-free evaluation has the potential to make machine translation evaluation substan- tially more scalable, allowing us to pivot eas- ily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual model can achieve state of the art results when used as a reference-free metric. We experiment with various modifica- tions to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weak- nesses of the approach, and find that it is sur- prisingly robust and likely to offer reasonable performance across a broad spectrum of do- mains and different system qualities. 1 Introduction Traditional automatic metrics for machine transla- tion (MT), such as BLEU (Papineni et al., 2002), score MT output by comparing it to one or more ref- erence translations. This has several disadvantages. First, high-quality reference translations are expen- sive to create. This means that in practice, evalu- ation is usually carried out with relatively small, carefully curated test corpora. The need for care- ful preparation limits the number of domains for which an MT system can be conveniently assessed, and small test-set sizes can make it difficult to draw robust conclusions (Card et al., 2020). Second, enshrining ground truth in a small number of refer- ences (usually just one) is inherently problematic, since valid translations can vary along many di- mensions; Freitag et al. (2020b) demonstrate that different (correct) references for the same test set can result in different system rankings according to the same reference-based metric. Finally, scoring the similarity between an MT hypothesis and a ref- erence translation involves recognizing the extent to which they are mutual paraphrases. When gross discrepancies exist, this is a relatively easy problem for which surface-level metrics can provide a reli- able signal, but capturing the subtle errors typical of high-quality MT is more difficult, and it is not clear whether it is substantially easier than scoring the similarity between texts in different languages. These problems can be avoided by looking only at the source text when assessing MT output. There is evidence that this is the best practice for human evaluation (Toral, 2020). Moreover, it has recently been investigated for automatic metrics as well (Yankovskaya et al., 2019; Lo, 2019; Zhao et al., 2020; Ma et al., 2019). Such reference-free metrics are flexible and scalable, but since they are essen- tially performing the same task as an MT model, they raise a circularity concern: if we can reliably score MT output, why wouldn’t we use the scoring model to produce better output? One answer to this is practical: the scoring model might …

Content truncated to 500 words