Unbabel’s Participation in the WMT17 Translation Quality Estimation Shared Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 569–574 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Unbabel’s Participation in the WMT17 Translation Quality Estimation Shared Task Andr´e F. T. Martins Unbabel & Instituto de Telecomunicac¸˜oes Lisbon, Portugal andre.martins@unbabel.com Fabio N. Kepler Unbabel University of Pampa, Alegrete, Brazil kepler@unbabel.com Jos´e L. Monteiro Unbabel Lisbon, Portugal jose@unbabel.com Abstract This paper presents the contribution of the Unbabel team to the WMT 2017 Shared Task on Translation Quality Estimation. We participated on the word-level and sentence-level tracks. We describe our two submitted systems: (i) STACKEDQE, a “pure” QE system, trained only on the provided training sets, which is a stacked combination of a feature-rich se- quential linear model with a neural net- work, and (ii) FULLSTACKEDQE, which also stacks the predictions of an automatic post-editing system, trained on additional data. When evaluated on the English- German and German-English datasets, FULLSTACKEDQE achieved word-level F MULT 1 scores of 56.6% and 52.9%, and sentence-level correlation Pearson scores of 64.1% and 62.6%, respectively. Our system ranked second in both tracks, be- ing statistically indistinguishable from the best system in the word-level track. 1 Introduction Quality estimation is the task of evaluating a trans- lation system’s quality without access to reference translations (Blatz et al., 2004; Specia et al., 2013). This paper describes the contribution of the Unba- bel team to the Shared Task on Sentence-Level and Word-Level Quality Estimation (QE Tasks 1 and 2) at the 2017 Conference on Statistical Machine Translation (WMT 2017). In the word-level task, the goal is to predict the word-level quality of machine translated text, by assigning a label of OK or BAD to each word in the translation. The sentence-level task attempts to predict the HTER of each sentence, along with a ranking of the sentences. Two language pairs and domains are considered: English-German (IT domain) and German-English (medical domain). Our submission is largely based on the approach that we have recently proposed in Martins et al. (2017), which ensembles a “pure” quality estima- tion system with predictions derived from an au- tomatic post-editing system. The focus was on developing a word-level system, and to use the word label predictions to predict the sentence- level HTER. Our system architecture is described in full de- tail in the following sections. We first describe our “pure” QE system (§2), which consists of a neural model (NEURALQE) stacked into a lin- ear feature-rich classifier (LINEARQE). Then, we train an APE system (using a large amount of arti- ficial “roundtrip translations”) and adapt it to pre- dict word-level quality labels (yielding APEQE, §3). We show that the pure and the APE-based QE system are highly complementary (§4): our best system is a stacked combination of LINEARQE, NEURALQE, and APEQE. By employing a simple word-to-sentence conversion, we adapt our sys- tems to sentence-level QE. Overall, we achieve word-level F MULT 1 scores of 56.6% and 52.9% and sentence-level Pearson scores of 64.1% and 62.6% for English-German and …

Content truncated to 500 words