Quality Estimation via Backtranslation at the WMT 2022 Quality Estimation Task

Quality Estimation via Backtranslation at the WMT 2022 Quality Estimation Task Sweta Agrawal∗ University of Maryland sweagraw@umd.edu Niloufar Salehi University of California, Berkeley nsalehi@berkeley.edu Nikita Mehandru* University of California, Berkeley nmehandru@berkeley.edu Marine Carpuat University of Maryland marine@umd.edu Abstract This paper describes submission to the WMT 1:2022 Quality Estimation shared task (Task sentence-level quality prediction, Zerva et al. (2022)). We follow a simple and intu- itive approach: estimating MT quality by au- tomatically back-translating hypotheses into the source language using a multilingual MT system. Using standard MT evaluation met- rics, we then compare the resulting backtrans- lation with the original source. We find that even the best-performing backtranslation-based scores perform substantially worse than super- vised QE systems, including the organizers’ baseline. However, combining backtranslation- based metrics with off-the-shelf QE scorers improves correlation with human judgments, suggesting that they can indeed complement a supervised QE system. 1 Introduction Sophisticated approaches to MT quality estimation (QE) based on large pre-trained models and care- ful training regimen have enabled great progress in recent years. However, when using online MT systems, such QE technology is not yet available to users and backtranslation provides an appeal- ingly simple strategy to estimate translation quality whether by humans or by automated sytems. Lay users often rely on backtranslation to assess MT quality in languages that they do not understand (Somers, 2005; Mehandru et al., 2022). As a result, from a user experience standpoint, using backtrans- lation for QE is easy to explain. Furthermore, with the increasing popularity of multilingual neural MT systems that can easily translate between multiple language pairs in any direction, backtranslations are very cheap to obtain, since they do not even re- quire training an auxiliary MT system in the reverse translation direction. However, the effectiveness of backtranslation for estimating the quality of MT remains unclear. ∗ equal contribution. In early rule-based and statistical MT systems, Somers (2005) shows that, when using automatic evaluation methods (e.g., BLEU), backtranslation cannot discriminate good MT systems from bad ones, nor between texts that are easy or hard to translate. This led him to conclude that “round trip translation [is] good for nothing”. Recently, Moon et al. (2020) revisited the use of backtransla- tion for QE with neural systems for MT and with embedding-based similarity metrics to enable a more sophisticated comparison of the backtransla- tion with the source. They obtained strong results on the WMT 2019 QE task, outperforming the YISI-2 metric (Lo, 2019) on system-level evalua- tions, but exhibited rather low correlations on the segment-level task which is more directly aligned with how humans use BT to gauge MT quality. The goal of our submission is to pitch a backtranslation-based QE score that can comple- ment state-of-the-art quality estimation systems in the controlled settings of the WMT shared task (Zerva et al., 2022) and understand its reliability as a sentence-level quality estimation technique. 2 Approach Following Moon et al. (2020), given a source sen- tence x and a MT hypothesis, we translate y back into the source …

Content truncated to 500 words