Machine Translation Reference-less Evaluation using YiSi-2 with Bilingual Mappings of Massive Multilingual Language Model

Proceedings of the 5th Conference on Machine Translation (WMT), pages 903–910 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 903 Machine Translation Reference-less Evaluation using YiSi-2 with Bilingual Mappings of Massive Multilingual Language Model Chi-kiu Lo and Samuel Larkin Multilingual Text Processing Digital Technologies Research Centre National Research Council Canada (NRC-CNRC) 1200 Montreal Road, Ottawa, ON K1A 0R6, Canada {chikiu.lo,samuel.larkin}@nrc-cnrc.gc.ca Abstract We present a study on using YiSi-2 with mas- sive multilingual pretrained language models for machine translation (MT) reference-less evaluation. Aiming at finding better seman- tic representation for semantic MT evaluation, we first test YiSi-2 with contextual embed- dings extracted from different layers of two dif- ferent pretrained models, multilingual BERT and XLM-RoBERTa. We also experiment with learning bilingual mappings that trans- form the vector subspace of the source lan- guage to be closer to that of the target language in the pretrained model to obtain more accu- rate cross-lingual semantic similarity represen- tations. Our results show that YiSi-2’s corre- lation with human direct assessment on trans- lation quality is greatly improved by replacing multilingual BERT with XLM-RoBERTa and projecting the source embeddings into the tar- get embedding space using a cross-lingual lin- ear projection (CLP) matrix learnt from a small development set. 1 Introduction The machine translation quality estimation as a metric (QE as a metric) task was first introduced in WMT 2019 (Ma et al., 2019; Fonseca et al., 2019) to encourage the exploration of reference-less eval- uation metrics. QE as a metric task shifts the use case of the QE systems from assisting professional translators to estimate post-editing efforts to assist- ing MT developers or general MT users to discrimi- nate the translation quality of different MT systems without the presence of a human reference transla- tion. YiSi-2, the reference-less variants of the YiSi metric (Lo, 2019), was the only metric who partic- ipated in evaluating all the translation directions in WMT 2019 QE as a metric shared task. The QE as a metric task is very similar to Task 1 (Sentence-level direct assessment) of WMT20’s quality estimation shared task where metric per- formance is evaluated in terms of correlation at the sentence-level with human direct assessment scores on translation quality. The subtle but cru- cial difference between the WMT20 QE Task 1 and the QE as a metric task is that QE systems for the former task is trained specifically to estimate the quality of a single MT system whereas QE met- rics for the latter task is generalized for multiple machine translation systems. The QE systems for WMT20’s QE Task 1 have access to the MT system that generate the translations while the reference- less metrics for the latter task have no information on the MT systems being evaluated. In WMT 2019 metrics shared task, pretrained multilingual BERT (Devlin et al., 2018) was used in YiSi for both MT reference-based (YiSi- 1) and reference-less (YiSi-2) evaluation in all tested translation directions where monolingual pretrained BERT model was not available for the target language …

Content truncated to 500 words