Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1068–1074 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1068 Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns Lei Zhou§ Liang Ding† Koichi Takeda§ §FVCRC, Graduate School of Informatics, Nagoya University {zhou.lei@a.mbox,takedasu@i}.nagoya-u.ac.jp †UBTECH Sydney AI Centre, School of Computer Science Faculty of Engineering, The University of Sydney ldin3097@uni.sydney.edu.au Abstract This paper describes our submission of the WMT 2020 Shared Task on Sentence Level Direct Assessment, Quality Estimation (QE). In this study, we empirically reveal the mismatching issue when directly adopting BERTScore (Zhang et al., 2020) to QE. Specif- ically, there exist lots of mismatching errors between source sentence and translated candi- date sentence with token pairwise similarity. In response to this issue, we propose to expose explicit cross lingual patterns, e.g. word align- ments and generation score, to our proposed zero-shot models. Experiments show that our proposed QE model with explicit cross-lingual patterns could alleviate the mismatching is- sue, thereby improving the performance. En- couragingly, our zero-shot QE method could achieve comparable performance with super- vised QE method, and even outperforms the supervised counterpart on 2 out of 6 directions. We expect our work could shed light on the zero-shot QE model improvement. 1 Introduction Translation quality estimation (QE) (Blatz et al., 2004; Specia et al., 2018, 2020) aims to pre- dict the quality of translation hypothesis without golden-standard human references, setting it apart from reference-based translation metrics. Existing reference-based evaluation metrics, e.g. BLEU (Pa- pineni et al., 2002), METEOR (Banerjee and Lavie, 2005), NIST (Doddington, 2002), ROUGE (Lin, 2004), TER (Snover et al., 2006), are commonly used in language generation tasks including transla- tion, summarization, and captioning but all heavily rely on the quality of given references. Recently, (Edunov et al., 2020) show that reference-based automatic evaluation metrics, e.g., BLEU, are not always reliable because the human translated references are translationese (Koppel and Назва ##но число го ##тов к##ящихся от ##пра ##в в##ке Дон ##бас ##с нов ##ран ##об цев из Украины The number of re ##c ##ruit ##s from Ukraine preparing to be sent to Don ##bas named has been 0.709 0.661 0.659 0.642 0.645 Назва ##но число го ##тов к##ящихся от ##пра ##в в##ке Дон ##бас ##с нов ##ран ##об цев из Украины The number of re ##c ##ruit ##s from Ukraine preparing to be sent to Don ##bas named has been 0.56 0.529 0.527 0.642 0.645 Figure 1: Example of mismatching error, Russian→English. On the left, token “Назва” is mismatched to “The” with the maximal probability (within the red rectangle) only. On the right, guided by our proposed cross-lingual patterns, “Назва” is correctly matched to the token “named” with the maximal probability (within the green rectangle.) Ordan, 2011; Graham et al., 2019). Thus, an auto- matic method with no access to any references, i.e., QE, is highly appreciated. In this paper, we mainly focus on sentence level QE metrics, where existing studies categorize it into two classes: 1) supervised QE with human assessment as supervision …

Content truncated to 500 words