RTM Ensemble Learning Results at Quality Estimation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 999–1003 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 999 RTM Ensemble Learning Results at Quality Estimation Task Ergun Bic¸ici ergun.bicici@boun.edu.tr Electrical and Electronics Engineering Department, Bo˘gazic¸i University orcid.org/0000-0002-2293-2031 Abstract We obtain new results using referential transla- tion machines (RTMs) with predictions mixed and stacked to obtain a better mixture of ex- perts prediction. We are able to achieve better results than the baseline model in Task 1 sub- tasks. Our stacking results significantly im- prove the results on the training sets but de- crease the test set results. RTMs can achieve to become the 5th among 13 models in ru-en subtask and 5th in the multilingual track of sentence-level Task 1 based on MAE. 1 Introduction Quality estimation task in WMT20 (Specia et al., 2020) (QET20) address machine translation (MT) performance prediction (MTPP), where transla- tion quality is predicted without using reference translations, at the sentence- (Tasks 1 and 2), word- (Task 2), and document-levels (Task 3). Task 1 predicts the sentence-level direct assess- ment (DA) in 7 language pairs categorized accord- ing to the MT resources available: • high-resource, English–German (en-de), English–Chinese (en-zh), and Russian- English (en-ru) • medium-resource, Romanian–English (ro- en) and Estonian–English (et-en), and • low-resource, Sinhalese–English (si-en) and Nepalese–English (ne-en). en-ru contains sentences from both Wikipedia and Reddit articles while others use only Wikipedia sentences with 7000 sentences for training, 1000 for development, and 1000 for testing. The target to predict in Task 1 is z-standardised DA scores, which changes the range from [0, 100] for DA scores to [3.178, −7.542] in z-standardized DA scores. RTM interpretants Task Train Test setting Training LM Task 1 (en-de) 8000 1000 bilingual 0.3 M 5 M Task 1 (en-zh) 8000 1000 monolingual en 0.2 M 3.5 M Task 1 (si-en) 8000 1000 monolingual en 0.2 M 3.5 M Task 1 (ne-en) 8000 1000 monolingual en 0.2 M 3.5 M Task 1 (et-en) 8000 1000 monolingual en 0.2 M 3.5 M Task 1 (ro-en) 8000 1000 monolingual en 0.2 M 3.5 M Task 1 (ru-en) 8000 1000 bilingual 0.2 M 4 M Task 2 (en-de) 8000 1000 bilingual 0.3 M 5 M Task 2 (en-zh) 8000 1000 monolingual en 0.2 M 3.5 M Table 1: Number of instances in the tasks and the size of the interpretants used. The target to predict in Task 2 is sen- tence HTER (human-targeted translation edit rate) scores (Snover et al., 2006) and binary classifica- tion of word-level translation errors. We partic- ipated in sentence-level subtasks, which include English-German and English-Chinese in Task 2. Table 1 lists the number of sentences in the train- ing and test sets for each task and the number of instances used as interpretants in the referential translation machine (RTM) (Bic¸ici, 2018; Bic¸ici and Way, 2015) models (M for million). We tokenize and truecase all of the corpora us- ing Moses’ (Koehn et al., 2007) processing tools.1 LMs are built using kenlm (Heafield et al., 2013). 2 …

Content truncated to 500 words