Predicting Translation Performance with Referential Translation Machines

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 540–544 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Predicting Translation Performance with Referential Translation Machines Ergun Bic¸ici orcid.org/0000-0002-2293-2031 bicici.github.com Abstract Referential translation machines achieve top performance in both bilingual and monolingual settings without accessing any task or domain specific information or resource. RTMs achieve the 3rd system re- sults for German to English sentence-level prediction of translation quality and the 2nd system results according to root mean squared error. In addition to the new fea- tures about substring distances, punctua- tion tokens, character n-grams, and align- ment crossings, and additional learning models, we average prediction scores from different models using weights based on their training performance for improved results. 1 Introduction Quality estimation task (QET) in WMT17 (Bo- jar et al., 2017) (QET17) is about prediction of the quality of machine translation output at the sentence- (Task 1), word- (Task 2), and phrase- level (Task 3) in IT and pharmaceutical domains without using reference translations. Prediction of translation performance can help in estimating the effort required for correcting the translations dur- ing post-editing by human translators if needed. RTMs are capable to model different domains and tasks while achieving top performance in both monolingual (Bic¸ici and Way, 2015) and bilingual settings (Bic¸ici, 2016b). We develop RTM mod- els for all of the three subtasks of QET17, which include English to German (en-de), and German to English (de-en) translation directions. Task 1 is about predicting HTER (human-targeted trans- lation edit rate) scores (Snover et al., 2006), Task 2 is about binary classification of word-level quality, Figure 1: RTM depiction: ParFDA selects inter- pretants close to the training and test data using parallel corpus in bilingual settings and mono- lingual corpus in the target language or just the monolingual target corpus in monolingual set- tings; an MTPPS use interpretants and training data to generate training features and another use interpretants and test data to generate test features in the same feature space; learning and prediction takes place taking these features as input. and Task 3 is about binary classification of phrase- level quality. 2 Referential Translation Machines Referential translation machine (RTM) models are predict data translation between the instances in the training set and the test set. RTMs use inter- pretants, data close to the task instances, to de- rive features measuring the closeness of the test sentences to the training data, the difficulty of translating them, and to identify translation acts between any two data sets for building predic- tion models. RTMs are applicable in different domains and tasks and in both monolingual and bilingual settings. Figure 1 depicts RTMs and explains the model building process. RTMs use ParFDA (Bic¸ici, 2016a) for instance selection and machine translation performance prediction sys- tem (MTPPS) (Bic¸ici and Way, 2015) for generat- 540 Task Model DeltaAvg rP rS RMSE MAE RAE MAER MRAER Rank Task 1 en-de MIX 4 8.64 0.4544 0.4768 0.1707 0.1296 0.8483 0.7594 0.7962 9 …

Content truncated to 500 words