Translation Quality Estimation by Jointly Learning to Score and Rank

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2592–2598, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2592 Translation Quality Estimation by Jointly Learning to Score and Rank Jingyi Zhang1 and Josef van Genabith1,2 1German Research Center for Artificial Intelligence (DFKI), Saarbrucken, Germany 2Department of Language Science and Technology, Saarland University, Germany Jingyi.Zhang@dfki.de,Josef.Van Genabith@dfki.de Abstract The translation quality estimation (QE) task, particularly the QE as a Metric task, aims to evaluate the general quality of a translation based on the translation and the source sen- tence without using reference translations. Su- pervised learning of this QE task requires hu- man evaluation of translation quality as train- ing data. Human evaluation of translation qual- ity can be performed in different ways, includ- ing assigning an absolute score to a translation or ranking different translations. In order to make use of different types of human evalua- tion data for supervised learning, we present a multi-task learning QE model that jointly learns two tasks: score a translation and rank two translations. Our QE model exploits cross- lingual sentence embeddings from pretrained multilingual language models. We obtain new state-of-the-art results on the WMT 2019 QE as a Metric task and outperform sentBLEU on the WMT 2019 Metrics task. 1 Introduction The translation quality estimation (QE) task (Fon- seca et al., 2019) aims to evaluate the quality of a translation based on the translation and the source sentence without using reference translations. The QE task includes word-level QE, sentence-level QE, document-level QE and QE as a Metric tasks. The QE as a Metric task requires QE models to score a translation on the sentence level similar to the sentence-level QE task, but these two tasks are different as the goal of the sentence-level QE task (Martins et al., 2017) is to predict the percentage of edits needed to fix the translation for post-editing purposes while the goal of the QE as a Metric task is to estimate the general quality of the translation like machine translation (MT) evaluation metrics, such as BLEU (Papineni et al., 2002) and Meteor (Denkowski and Lavie, 2014), except without using reference translations. Supervised learning of the QE as a Metric task requires human evaluation of translation quality as training data. Human evaluation of translation qual- ity is generally very costly and can be performed in different ways, such as Direct Assessment (DA: requiring human assessors to assign an absolute score to a translation) (Barrault et al., 2019; Gra- ham et al., 2013, 2014, 2017) or Relative Ranking (RR: requiring human assessors to rank different translations) (Bojar et al., 2015). Since the QE as a Metric task requires QE models to assign an absolute score to a translation, DA human evalu- ation data can be straightforwardly used as train- ing data for the QE as a Metric task. In order to also make use of the RR human evaluation data, we propose a multi-task learning QE model that jointly learns two tasks, score a translation and …

Content truncated to 500 words