Return to the Source: Assessing Machine Translation Suitability

Return to the Source: Assessing Machine Translation Suitability Francesco Fernicola1,2, Silvia Bernardini1, Federico Garcea1 Adriano Ferraresi1 and Alberto Barron-Cedeño1 1 Università di Bologna, Forlì, Italy 2 Institute for Applied Linguistics, Eurac Research, Bolzano/Bozen, Italy [francesco.fernicola2, silvia.bernardini, federico.garcea2]@unibo.it [adriano.ferraresi, a.barron]@unibo.it Abstract We approach the task of assessing the suitability of a source text for translation by transferring the knowledge from estab- lished MT evaluation metrics to a model able to predict MT quality a priori from the source text alone. To open the door to experiments in this regard, we depart from reference English–German parallel corpora to build a corpus of 14,253 source text–quality score tuples. The tuples in- clude four state-of-the-art metrics: cush- LEPOR, BERTScore, COMET, and Tran- sQuest. With this new resource at hand, we fine-tune XLM-RoBERTa, both in a single-task and a multi-task setting, to predict these evaluation scores from the source text alone. Results for this method- ology are promising, with the single-task model able to approximate well-established MT evaluation and quality estimation met- rics —without looking at the actual ma- chine translations— achieving low Root Mean Square Error values in the [0.1–0.2] range and Pearson’s correlation scores up to 0.688. 1 Introduction There are many factors in play when assessing the suitability of a text for machine translation (MT). Readability might account for part of the problem, but the metrics designed for its estimation aim at assessing the level of education necessary to un- derstand a given text, from a monolingual perspec- tive (Gunning, 1969). As evidenced by Vanroy et © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. al. (2019), there is a clear-cut distinction between translatability, “the difficulty of a translation task”, and readability, “the difficulty of a monolingual text”. They argue that, although the two might overlap in some regards, a translation task cannot be solely defined based on monolingual features. Their study is centred on human translation (HT), but given that MT and post-editing (PE) represent the strongest future trend for both industry and academia, according to the latest ELIS language industry report (European Language Industry Sur- vey Research, 2022), our work seeks to advance the discussion in the field of MT. In fact, although quality improvements over the last few years have indeed been significant, the translation world has expressed a need, time and time again, for new methods and technologies to properly assess its quality (Kocmi et al., 2021). Most of the previous work in this regard has focused on the target translation; both in the reference-based machine translation evaluation (MTE), where the machine-translated segment is compared against a human reference, and in the more recent quality estimation techniques (QE), where the machine- translated segment is evaluated without any refer- ence (Freitag et al., 2021; Specia et al., 2021). This paper seeks a different perspective, switch- ing the focus to the source text, to assess whether a given segment will produce a high quality machine translation. We define this …

Content truncated to 500 words