Estimation vs Metrics: is QE Useful for MT Model Selection?

Estimation vs Metrics: is QE Useful for MT Model Selection? Anna Zaretskaya Jos´e Conceic¸˜ao Frederick Bane TransPerfect Passeig de Gr`acia, 11, Esc B 5-2 08007 Barcelona, Spain {azaretskaya, jconceicao, fbane}@translations.com Abstract This paper presents a case study of apply- ing machine translation quality estimation (QE) for the purpose of machine transla- tion (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation met- rics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for transla- tion industry such as risk prevention. 1 Introduction Machine translation quality estimation (QE) is a technique for predicting machine translation (MT) quality (Specia et al., 2009). As MT becomes the dominant tool in the translation industry, accurate estimation of the quality of MT output would be of great benefit to many business concerns such as budget allocation for human post-editing, esti- mating the usefulness of the MT output for gisting purposes, and selecting the best MT system out of a selection of systems. In addition to that, a re- liable QE model would also help linguists make more efficient use of their time. As opposed to MT evaluation, where MT out- put is compared to one or several human reference translations, QE attempts to perform the much more challenging task of predicting MT quality in the absence of a reference translation. QE can be performed on a word, sentence or document level, c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. and the output of a QE system is typically a score that is intended to correlate with a certain auto- matic or human MT evaluation metric. In this paper we present a case study where QE models were applied for the purpose of rank- ing different MT engines for a given document or text corpus. Our approach is based on obtain- ing segment-level QE scores for all segments in the document/corpus and using the average to se- lect the best MT system. We use two QE sys- tems to score the output of different MT engines and compare the results of the QE model with several automatic MT evaluation metrics, which include the post-editing distance (PED), HTER, BLEU score (Papineni et al., 2002), and weighted and unweighted sentence embedding similarity, as well as with human MT evaluation scores. We conclude the paper by reflecting on the usefulness of QE for MT engine selection, possible improve- ments to QE, and the limitations of our model and method. 2 Background and Motivation The main motivation of exploring the QE method for MT model selection is the fact that we are of- ten faced with a scenario where we need to choose the best MT system without a reference transla- tion. The only existing method for this is …

Content truncated to 500 words