Consistent Human Evaluation of Machine Translation across Language Pairs

Consistent Human Evaluation of Machine Translation across Language Pairs Daniel Licht META AI dlicht@fb.com Cynthia Gao META AI cynthiagao@fb.com Janice Lam META AI janilam@fb.com Francisco Guzm´an META AI fguzman@fb.com Mona Diab∗META AI mdiab@fb.com Philipp Koehn∗META AI / Johns Hopkins University phi@jhu.edu Abstract Obtaining meaningful quality scores for machine translation systems through human evaluation remains a challenge given the high variability between human evaluators, partly due to subjec- tive expectations for translation quality for different language pairs. We propose a new metric, XSTS, that is more focused on semantic equivalence. Moreover, we introduce a cross-lingual calibration method that enables more consistent assessment. We demonstrate the effectiveness of these novel contributions in large scale evaluation studies across up to 14 language pairs, with translation both into and out of English. 1 Introduction While machine translation systems are typically evaluated with automatic metrics like BLEU (Papineni et al., 2001), the gold standard for quality assessment is evaluation of machine trans- lation output by human evaluators. In fact, the validity of automatic metrics is justified by correlation to human evaluations. However, in practice individual human evaluators apply very different standards when assessing machine translation output, depending on their expectation of translation quality, their exposure to machine translation output, their language abilities, the presentation of source or reference translation, and vague category descriptions like ”mostly correct”. This is especially a problem when the goal is to obtain meaningful scores across language pairs, to assess, for instance, if a machine translation system for any given language pair is of sufficiently high quality to be put to use. We address this problem of high variability and cross-lingual consistency by two novel contributions: (1) a new scoring metric XSTS that is focused on meaning; and, (2) an evaluation protocol that allows for calibration of scores across evaluators and across language pairs. Our studies show that the XSTS score yields higher inter-annotator agreement compared against a 5-Point Raw Scale. We also show that our calibration leads to improved correlation of system scores to our subjective expectations of quality based on linguistic and resource aspects as well as improved correlation with automatic scores. ∗Equal contribution as senior authors. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 309 2 Related Work The DARPA evaluation of the 1990s tasked human evaluators to assign scores from 1 to 5 to judge the fluency and adequacy of translations (White and O’Connell, 1996), with vague definitions like much meaning for an adequacy score of 3 or the slightly offensive non-native English for a fluency score of 3. This scale was also used in the first human evaluation of the Workshop on Statistical Machine Translation (WMT) (Koehn and Monz, 2006). Note that these evaluations aim at a different goal than the one we are concerned with here: their main purpose is to rank the output of different machine translation systems against one another — without the need to report a …

Content truncated to 500 words