Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1311–1325 May 2-6, 2023 ©2023 Association for Computational Linguistics Poor Man’s Quality Estimation: Predicting Reference-Based MT Metrics Without the Reference Vilém ZouharE Shehzaad DhuliawalaE Wangchunshu ZhouE Nico DaheimD Tom KocmiM Yuchen Eleanor JiangE Mrinmaya SachanE EETH Zürich MMicrosoft DTU Darmstadt {vzouhar,msachan}@ethz.ch Abstract Machine translation quality estimation (QE) predicts human judgements of a translation hy- pothesis without seeing the reference. State- of-the-art QE systems based on pretrained lan- guage models have been achieving remarkable correlations with human judgements yet they are computationally heavy and require human annotations, which are slow and expensive to create. To address these limitations, we define the problem of metric estimation (ME) where one predicts the automated metric scores also without the reference. We show that even with- out access to the reference, our model can es- timate automated metrics (ρ=60% for BLEU, ¯ρ=51% for other metrics) at the sentence-level. Because automated metrics correlate with hu- man judgements, we can leverage the ME task for pre-training a QE model. For the QE task, we find that pre-training on TER is better (ρ=23%) than training for scratch (ρ=20%). github.com/zouharvi/mt-metric-estimation 1 Introduction Quality estimation (QE) is often used in machine translation (MT) production pipelines where we need to make decisions based on the quality of an MT output but where the reference is unavailable (Specia et al., 2020, 2021). For example, QE is used in translation companies to decide whether to send a specific MT output for post-editing to a human translator or whether to use it directly (Tamchyna, 2021; Zouhar et al., 2021; Murgolo et al., 2022). In this scenario, an accurate QE system has the potential to save expensive trans- lator effort. However, training QE models usu- ally requires human-annotated judgements of the translation quality (Specia et al., 2013; Rubino and Sumita, 2020; Rei et al., 2020a). These human annotations are scarce and costly, especially for Source text Part 1: large data Reference MT Hypothesis predict Metric Estimator (pretraining) Automated metric score Source text Part 2: limited data Reference Hypothesis predict Quality Estimator (fine-tuning) Human scores MT Figure 1: Pipeline for pre-training on automated metrics (top) and fine-tuning on limited quality estimation data (bottom). Dotted lines are optional dependency. low-resource language directions, and may need to be replicated for new MT systems and domains (de Souza et al., 2014) We investigate if automated MT metrics can be used to reduce the cost of learning QE models. Automated metrics can be run with no additional costs and can be used to generate large amounts of QE data. If these metrics correlate well with human judgements, this larger data can be used as a partial substitute for human data during training. Data augmentation and synthetic QE data via automated metrics has already been explored (Heo et al., 2021; Baek et al., 2020; Cui et al., 2021), though never in the pre-training & fine-tuning fashion. Our work is guided by a simple intuition. In- …

Content truncated to 500 words