QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation Julia Kreutzer and Shigehiko Schamoni Computational Linguistics Heidelberg University 69120 Heidelberg, Germany {kreutzer,schamoni}@cl.uni-heidelberg.de Stefan Riezler Computational Linguistics & IWR Heidelberg University 69120 Heidelberg, Germany riezler@cl.uni-heidelberg.de Abstract This paper describes the system submit- ted by the University of Heidelberg to the Shared Task on Word-level Quality Esti- mation at the 2015 Workshop on Statis- tical Machine Translation. The submit- ted system combines a continuous space deep neural network, that learns a bilin- gual feature representation from scratch, with a linear combination of the manually defined baseline features provided by the task organizers. A combination of these orthogonal information sources shows sig- nificant improvements over the combined systems, and produces very competitive F1-scores for predicting word-level trans- lation quality. 1 Introduction This paper describes the University of Heidel- berg submission to the Shared Task on Word- level Quality Estimation (QE Task 2) at the 2015 Workshop on Statistical Machine Transla- tion (WMT15). The task consists of predict- ing the word-level quality level (“OK”/“BAD”) of English-to-Spanish machine translations, without the use of human references, and without insight into the translation derivations, that is, by treat- ing the Machine Translation (MT) system that pro- duced the translations as a black box. The task organizers provided training and de- velopment data comprising tokenized MT outputs that were automatically annotated for errors as edit operations (replacements, insertions, or deletions) with respect to human post-edits (Snover et al., 2006). Furthermore, a set of 25 baseline features that operate on source and target translation, but do not use features of the SMT pipeline that pro- duced the translations, was provided. Even though the distribution of binary labels is skewed towards “OK” labels, even more so than in the previous QE task at WMT141, the most common approach is to treat the problem as a supervised classifica- tion task. Furthermore, most approaches rely on manually designed features, including source and target contexts, alignments, and generalizations by linguistic categories (POS, syntactic dependency links, WordNet senses) as reported by Bojar et al. (2014), similar to the 25 feature templates pro- vided by the organizers. We apply the framework of Collobert et al. (2011) to learn bilingual correspondences “from scratch”, i.e. from raw input words. To this aim, a continuous space deep neural network is pre-trained by initializing the lookup-table with distributed word representations (Mikolov et al., 2013b), and fine-tuned for the QE classification task by back-propagating word-level prediction er- rors using stochastic gradient descent (Rumelhart et al., 1986). Moreover, we train a linear combi- nation of the manually defined baseline features provided by the task organizers. A combination of the orthogonal information based on the continu- ous space features and the manually chosen base- line features shows significant improvements over the combined systems, and produces very compet- itive F1 scores for predicting word-level transla- tion quality. 2 Deep …

Content truncated to 500 words