UGENT-LT3 SCATE System for Machine Translation Quality Estimation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 353–360, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. UGENT-LT3 SCATE System for Machine Translation Quality Estimation Arda Tezcan Veronique Hoste Bart Desmet Lieve Macken Department of Translation, Interpreting and Communication Ghent University Groot-Brittanniëlaan 45, 9000 Ghent, Belgium {arda.tezcan, veronique.hoste, bart.desmet, lieve.macken}@ugent.be Abstract This paper describes the submission of the UGENT-LT3 SCATE system to the WMT15 Shared Task on Quality Estima- tion (QE), viz. English-Spanish word and sentence-level QE. We conceived QE as a supervised Machine Learning (ML) problem and designed additional features and combined these with the baseline feature set to estimate quality. The sen- tence-level QE system re-uses the word level predictions of the word-level QE system. We experimented with different learning methods and observe improve- ments over the baseline system for word- level QE with the use of the new features and by combining learning methods into ensembles. For sentence-level QE we show that using a single feature based on word-level predictions can perform better than the baseline system and using this in combination with additional features led to further improvements in performance. 1 Introduction Machine Translation (MT) Quality Estimation (QE) is the task of providing a quality indicator for unseen automatically translated sentences without relying on reference translations (Gandrabur & Foster, 2003; Blatz et al., 2004). Predicting the quality of MT output has many applications in computer-aided translation work- flows that utilize MT, including error analysis (Ueffing and Ney 2007), filtering translations for human post-editing (Specia et al., 2009) and comparing the quality of different MT systems (Rosti et al. 2007). The most common approach is to treat the QE problem as a supervised Machine Learning (ML) task, using standard regression or classification algorithms. A considerable amount of related work on both word and sentence-level QE is de- scribed in the WMT shared tasks of previous years (Bojar et al., 2014; Bojar et al., 2013). The WMT 2015 QE shared task proposes three evaluation tasks: (1) scoring and ranking sentences according to predicted post-editing effort given a source sentence and its translation; (2) predicting the individual words that require post-editing; and (3) predicting the quality at document level. In this paper, we describe the UGENT-LT3 SCATE submissions to task 1 (sentence-level QE) and task 2 (word-level QE). Sentence-level and word-level QE are related tasks. Sentence-level QE assigns a global score to an automatically translated sentence whereas word-level QE is more fine-grained and tries to detect the problematic word sequences. There- fore we first developed a word-level QE system and incorporate the word-level predictions as additional features in the sentence-level QE sys- tem. The usefulness of including word-level pre- dictions in sentence-level QE has already been demonstrated by de Souza et al. (2014) For both tasks, we extracted additional fea- tures and combine these with the baseline feature set to estimate quality. The new features try to capture either accuracy or fluency errors, where accuracy is concerned with how much of the meaning expressed in …

Content truncated to 500 words