MT Quality Estimation: The CMU System for WMT‘13

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 373–379, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics MT Quality Estimation: The CMU System for WMT’13 Almut Silja Hildebrand Carnegie Mellon University Pittsburgh, USA silja@cs.cmu.edu Stephan Vogel Qatar Computing Research Institute Doha, Qatar svogel@qf.org.qa Abstract In this paper we present our entry to the WMT’13 shared task: Quality Estima- tion (QE) for machine translation (MT). We participated in the 1.1, 1.2 and 1.3 sub-tasks with our QE system trained on features from diverse information sources like MT decoder features, n-best lists, mono- and bi-lingual corpora and giza training models. Our system shows com- petitive results in the workshop shared task. 1 Introduction As MT becomes more and more reliable, more people are inclined to use automatically translated texts. If coming across a passage that is obviously a mistranslation, any reader would probably start to doubt the reliability of the information in the whole article, even though the rest might be ad- equately translated. If the MT system had a QE component to mark translations as reliable or pos- sibly erroneous, the reader would know to use in- formation from passages marked as bad transla- tions with caution, while still being able to trust other passages. In post editing a human translator could use translation quality annotation as an indi- cation to whether editing the MT output or trans- lating from scratch might be faster. Or he could use this information to decide where to start in or- der to improve the worst passages first or skip ac- ceptable passages altogether in order to save time. Confidence scores can also be useful for applica- tions such as cross lingual information retrieval or question answering. Translation quality could be a valuable ranking feature there. Most previous work in the field estimates con- fidence on the sentence level (e.g. Quirk et al. (2004)), some operate on the word level (e.g. Ueffing and Ney (2007), Sanchis et al. (2007), and Bach et al. (2011)), whereas Soricut and Echi- habi (2010) use the document level. Various classifiers and regression models have been used in QE in the past. Gandrabur and Foster (2003) compare single layer to Multi Layer Per- ceptron (MLP), Quirk et al. (2004) report that Lin- ear Regression (LR) produced the best results in a comparison of LR, MLP and SVM, Gamon et al. (2005) use SVM, Soricut and Echihabi (2010) find the M5P tree works best among a number of regression models, while Bach et al. (2011) define the problem as a word sequence labeling task and use MIRA. The QE shared task was added to the WMT evaluation campaign in 2012 (Callison-Burch et al., 2012), providing standard training and test data for system development. 2 WMT’13 Shared Task In this WMT Shared Task for Quality Estima- tion1 there were tasks for sentence and word level QE. We participated in all sub-tasks for Task 1: Sentence-level QE. Task 1.1: Scoring and ranking for post-editing effort focuses on predicting HTER …

Content truncated to 500 words