Referential Translation Machines for Quality Estimation

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 343–351, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Referential Translation Machines for Quality Estimation Ergun Bic¸ici Centre for Next Generation Localisation, Dublin City University, Dublin, Ireland. ergun.bicici@computing.dcu.ie Abstract We introduce referential translation ma- chines (RTM) for quality estimation of translation outputs. RTMs are a computa- tional model for identifying the translation acts between any two data sets with re- spect to a reference corpus selected in the same domain, which can be used for esti- mating the quality of translation outputs, judging the semantic similarity between text, and evaluating the quality of student answers. RTMs achieve top performance in automatic, accurate, and language inde- pendent prediction of sentence-level and word-level statistical machine translation (SMT) quality. RTMs remove the need to access any SMT system specific informa- tion or prior knowledge of the training data or models used when generating the trans- lations. We develop novel techniques for solving all subtasks in the WMT13 qual- ity estimation (QE) task (QET 2013) based on individual RTM models. Our results achieve improvements over last year’s QE task results (QET 2012), as well as our previous results, provide new features and techniques for QE, and rank 1st or 2nd in all of the subtasks. 1 Introduction Quality Estimation Task (QET) (Callison-Burch et al., 2012; Callison-Burch et al., 2013) aims to de- velop quality indicators for translations and pre- dictors without access to the references. Predic- tion of translation quality is important because the expected translation performance can help in esti- mating the effort required for correcting the trans- lations during post-editing by human translators. Bicici et al. (2013) develop the Machine Trans- lation Performance Predictor (MTPP), a state-of- the-art, language independent, and SMT system extrinsic machine translation performance predic- tor, which achieves better performance than the competitive QET baseline system (Callison-Burch et al., 2012) by just looking at the test source sen- tences and becomes the 2nd overall after also look- ing at the translation outputs in QET 2012. In this work, we introduce referential translation machines (RTM) for quality estimation of transla- tion outputs, which is a computational model for identifying the acts of translation for translating between any given two data sets with respect to a reference corpus selected in the same domain. RTMs reduce our dependence on any task depen- dent resource. In particular, we do not use the baseline software or the SMT resources provided with the QET 2013 challenge. We believe having access to glass-box features such as the phrase ta- ble or the n-best lists is not realistic especially for use-cases where translations may be provided by different MT vendors (not necessarily from SMT products) or by human translators. Even the prior knowledge of the training corpora used for build- ing the SMT models or any other model used when generating the translations diverges from the goal of independent and unbiased prediction of trans- lation quality. Our results show that we do not need to …

Content truncated to 500 words