Learning from Past Mistakes: Quality Estimation from Monolingual Corpora and Machine Translation Learning Stages

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 84–98 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 84 Learning from Past Mistakes: Quality Estimation from Monolingual Corpora and Machine Translation Learning Stages Thierry Etchegoyhen1 tetchegoyhen@vicomtech.org David Ponce1,2 adponce@vicomtech.org 1 Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) 2 University of the Basque Country - UPV/EHU Abstract Quality Estimation (QE) of Machine Translation output suffers from the lack of annotated data to train supervised models across domains and language pairs. In this work, we describe a method to generate synthetic QE data based on Neural Machine Translation (NMT) models at different learning stages. Our approach consists in training QE models on the errors pro- duced by different NMT model checkpoints, obtained during the course of model training, under the assumption that gradual learning will induce errors that more closely resemble those produced by NMT models in adverse conditions. We test this approach on English-German and Romanian-English WMT QE test sets, demonstrating that pairing translations from ear- lier checkpoints with translations of converged models outperforms the use of reference human translations and can achieve competitive results against human-labelled data. We also show that combining post-edited data with our synthetic data yields to significant improvements across the board. Our approach thus opens new possibilities for an efficient use of monolingual cor- pora to generate quality synthetic QE data, thereby mitigating the data bottleneck. 1 Introduction Significant improvements have been achieved in Machine Translation (MT) in recent years, in particular with the advent of Neural Machine Translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). However, the quality of automated translations can vary significantly depending on training data volumes, domain of application, language pairs or the complexity of specific source segments. Machine translation errors can significantly increase the risks and costs of using MT and the automatic estimation of MT quality becomes increasingly necessary to pinpoint or discard erroneous automatic translations. Traditionally, the quality of MT output has been assessed against human references, via automated metrics such as BLEU (Papineni et al., 2002) or TER (Snover et al., 2006). However, such references are not always available and are costly to produce, which has led to the devel- opment of Quality Estimation (QE) approaches based on the sole properties of the source and machine-translated sentences (Blatz et al., 2004; Specia et al., 2010). Most approaches to QE are based on supervised learning, traditionally via feature engineering (Specia et al., 2013), and, in recent years via neural models (Kim and Lee, 2016; Kim et al., 2017; Fonseca et al., 2019; 85 Specia et al., 2021). Although they provide the most accurate estimates to date, supervised methods depend on human annotations or post-edited translations to perform the task. The cost of producing quality QE training datasets hinders the development of QE models for the large number of possible domains and language pairs. Two …

Content truncated to 500 words