PE effort and neural-based automatic MT metrics: do they correlate?

PE effort and neural-based automatic MT metrics: do they correlate? Sergi Alvarez-Vidal Universitat Oberta de Catalunya salvarezvid@uoc.edu Antoni Oliver Universitat Oberta de Catalunya aoliverg@uoc.edu Abstract Neural machine translation (NMT) has shown overwhelmingly good results in re- cent times. This improvement in quality has boosted the presence of NMT in nearly all fields of translation. Most current trans- lation industry workflows include post- editing (PE) of MT as part of their process. For many domains and language combi- nations, translators post-edit raw machine translation (MT) to produce the final docu- ment. However, this process can only work prop- erly if the quality of the raw MT output can be assured. MT is usually evaluated using automatic scores, as they are much faster and cheaper. However, traditional auto- matic scores have not been good quality indicators and do not correlate with PE ef- fort. We analyze the correlation of each of the three dimensions of PE effort (tempo- ral, technical and cognitive) with COMET, a neural framework which has obtained outstanding results in recent MT evalua- tion campaigns. 1 Introduction In the last decade, MT has steadily increased its presence in all fields of translation. This is mainly due to the improvements in quality following the advances in NMT. Results of a recent language survey identify post-editing as the second most de- manded task among language providers and the activity with the highest growth potential, 64% © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. (ELIS, 2022). For many language combinations, translators edit, modify and correct the raw MT output to produce a final version. However, this process can only work properly if the quality of the raw MT output can be assured. To assess the quality of the MT output both man- ual and automatic metrics are currently used. On the one hand, manual evaluations include sentence ranking, fluency and adequacy, direct assessment (DA) (Graham et al., 2016), and explicit error anal- ysis, such as the ones based on the Multidimen- sional Quality Metrics (MQM) framework (Fre- itag et al., 2021a). Even though most of these evaluations produce quite reliable metrics, they have a high cost in time and resources (Papineni et al., 2002), which makes it complicated to use in a daily basis to assess the quality of MT sys- tems. They also suffer from low inter- and intra- annotator agreements (Snover et al., 2006). On the other hand, automatic evaluations pro- duce quick results. Even though these metrics were originally conceived as a way to compare two systems, in most scenarios they are used as the only means to assess the quality of an MT engine. Automatic scores usually show correla- tion with human judgments of translation (Cough- lin, 2003), even though they have been frequently questioned as a way to assess MT output (Mathur et al., 2020a), especially when they are used to compare high-quality systems (Ma et al., 2019). The most usual automatic metrics currently used, …

Content truncated to 500 words