BERGAMOT-LATTE Submissions for the WMT20 Quality Estimation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1010–1017 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1010 BERGAMOT-LATTE Submissions for the WMT20 Quality Estimation Shared Task Marina Fomicheva,1* Shuo Sun,2* Lisa Yankovskaya,3* Fr´ed´eric Blain,1 Vishrav Chaudhary,5 Mark Fishel,3 Francisco Guzm´an,5 Lucia Specia1,4 1University of Sheffield, 2Johns Hopkins University, 3University of Tartu, 4Imperial College London, 5Facebook AI 1{m.fomicheva,f.blain,l.specia}@sheffield.ac.uk 2ssun32@jhu.edu 3{lisa.yankovskaya,fishel}@ut.ee 5{fguzman,vishrav}@fb.com Abstract This paper presents our submission to the WMT2020 Shared Task on Quality Estimation (QE)1. We participate in Task 1 and Task 2 focusing on sentence-level prediction. We explore (a) a black-box approach to QE based on pre-trained representations; and (b) glass- box approaches that leverage various indica- tors that can be extracted from the neural MT systems. In addition to training a feature-based regression model using glass-box quality indi- cators, we also test whether they can be used to predict MT quality directly with no super- vision. We assess our systems in a multi- lingual setting and show that both types of approaches generalise well across languages. Our black-box QE models tied for the winning submission in four out of seven language pairs in Task 1, thus demonstrating very strong performance. The glass-box approaches also performed competitively, representing a light- weight alternative to the neural-based models. 1 Introduction Quality Estimation (QE) (Blatz et al., 2004; Specia et al., 2009) is an important part of Machine Trans- lation (MT) pipeline. It allows us to evaluate how good a translation is without comparison to refer- ence sentences. As part of the WMT20 Shared Task on Quality Estimation, two sentence-level tasks were proposed. In Task 1, participants are asked to predict human judgements of MT quality gen- erated following a methodology similar to Direct Assessment (DA) (Graham et al., 2017). The goal of Task 2 is to estimate the post-editing effort required in order to correct the MT outputs and measured using the HTER metric (Snover et al., 2006). 1http://www.statmt.org/wmt20/ quality-estimation-task.html ∗Equal contribution. This year’s task is different from the previous years in two important aspects: (i) the data includes seven language pairs, which are very different both typologically and in terms of translation quality; and (ii) the participants were provided with neural MT (NMT) models that were used for translation. We take advantage of this set up to compare black- box and glass-box approaches to QE. Furthermore, we test both approaches in a multilingual setting. The rest of this paper is organised as follows. Section 2 describes the glass-box (2.1) and black- box (2.2) QE methods that we explore in our sub- missions. Section 3 describes the dataset used for the WMT2020 Shared Task on Quality Estima- tion. Section 4 provides our experimental settings, whereas Section 5 presents the results. Conclusions are given in Section 6. 2 Approach Below we first describe our glass-box submissions based on the quality indicators that can be obtained as a by-product of decoding with an NMT system. Second, we present our neural-based QE submis- sions, which explore transfer learning with pre- …

Content truncated to 500 words