RTM Super Learner Results at Quality Estimation Task

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 885–889 November 10–11, 2021. ©2021 Association for Computational Linguistics 885 RTM Super Learner Results at Quality Estimation Task Ergun Bic¸ici orcid.org/0000-0002-2293-2031 bicici.github.io Abstract We obtain new results using referential transla- tion machines (RTMs) with predictions mixed to obtain a better mixture of experts prediction. Our super learner results improve the results and provide a robust combination model. 1 Introduction Quality estimation task in WMT21 (Specia et al., 2021) (QET21) address machine translation (MT) performance prediction (MTPP), where transla- tion quality is predicted without using reference translations, at the sentence-level (Tasks 1, 2, and 3) and with classification of sentences into con- taining a critical error or not (Task 3). Task 1 pre- dicts the sentence-level direct assessment (DA) in 11 language pairs categorized according to the MT resources available: • high-resource, English–German (en-de), English–Chinese (en-zh), and Russian- English (en-ru), • medium-resource, Romanian–English (ro- en) and Estonian–English (et-en), • low-resource, Sinhalese–English (si-en) and Nepalese–English (ne-en), and • no-resource, English–Czech (en-cs), English–Japanese (en-ja), Pashto–English (ps-en), and Khmer–English (km-en) for zero-shot prediction. en-ru contains sentences from both Wikipedia and Reddit articles while others use only Wikipedia sentences with 7000 sentences for training, 1000 for development, 1000 for test QET in 2020, and 1000 for testing at QET21. The target to pre- dict in Task 1 is z-standardised DA scores, which changes the range from [0, 100] for DA scores to [3.178, −7.542] in z-standardized DA scores. RTM interpretants Task Train Test setting Training LM Task 1 and Task 2 en-de 9000 1000 bilingual 0.3 M 3.5 M en-zh 9000 1000 bilingual 0.2 M 3.5 M et-en 9000 1000 bilingual 0.2 M 3.5 M ne-en 9000 1000 bilingual 0.2 M 3.5 M ro-en 9000 1000 bilingual 0.2 M 3.5 M ru-en 9000 1000 bilingual 0.2 M 3.5 M si-en 9000 1000 bilingual 0.2 M 3.5 M en-cs 63000 1000 bilingual 0.2 M 3.5 M en-ja 63000 1000 bilingual 0.2 M 3.5 M km-en 63000 1000 bilingual 0.2 M 3.5 M ps-en 63000 1000 bilingual 0.2 M 3.5 M Task 3 en-cs 9000 1000 bilingual 0.2 M 3.5 M en-de 9000 1000 bilingual 0.2 M 3.5 M en-ja 9000 1000 bilingual 0.2 M 3.5 M en-zh 9000 1000 bilingual 0.2 M 3.5 M Table 1: Number of instances in the tasks and the size of the interpretants used. The target to predict in Task 2 is sen- tence HTER (human-targeted translation edit rate) scores (Snover et al., 2006). We participated in sentence-level subtasks. Table 1 lists the num- ber of sentences in the training and test sets for each task and the number of instances used as interpretants in the referential translation ma- chine (RTM) (Bic¸ici and Way, 2015; Bic¸ici, 2020) models (M for million). In zero-shot predic- tion, we use all of the training instances made available to the task in all 7 translation direc- tions. We tokenize and truecase all of the cor- pora using Moses’ (Koehn et al., 2007) process- ing tools.1 …

Content truncated to 500 words