Black Box Features for the WMT 2012 Quality Estimation Shared Task

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 91–95, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Black Box Features for the WMT 2012 Quality Estimation Shared Task Christian Buck School of Informatics University of Edinburgh Edinburgh, UK, EH8 9AB christian.buck@ed.ac.uk Abstract In this paper we introduce a number of new features for quality estimation in machine translation that were developed for the WMT 2012 quality estimation shared task. We find that very simple features such as indicators of certain characters are able to outperform com- plex features that aim to model the connection between two languages. 1 Introduction and Task This paper describes the features and setup used in our submission to the WMT 2012 quality estimation (QE) shared task. Given a machine translation (MT) system and a corpus of its translations which have been rated by humans, the task is to build a predic- tor that can accurately estimate the quality of fur- ther translations. The human ratings range from 1 (incomprehensible) to 5 (perfect translation) and are given as the mean rating of three different judges. fJFormally we are presented with a source sentence 1 and a translation eI 1 and we need to assign a score S(fJ 1 , eI 1) ∈ [1, 5] or, in the ranking task, order the source-translation pairs by expected quality. 2 Resources The organizers have made available a baseline QE system that consists of a number of well established features (Blatz et al., 2004) and serves as a starting point for development. Furthermore the MT system that generated the translations is available along with its training data. Compared to the large training cor- pus of the MT engine, the QE system is based on a much smaller training set as detailed in Table 1. # sentences europarl-nc 1,714,385 train 1,832 test 422 Table 1: Corpus statistics 3 Features In the literature (Blatz et al., 2004) a large number of features have been considered for confidence es- timation. These can be grouped into four general categories: 1. Source features make a statement about the source sentence, assessing the difficulty of translating a particular sentence with the sys- tem at hand. Some sentences may be very easy to translate, e.g. short and common phrases, while long and complex sentences are still be- yond the system’s capabilities. 2. Translation features model the connection be- tween source and target. While this is very closely related to the general problem of ma- chine translation, the advantage in confidence estimation is that we can exercise unconstruc- tive criticism, i.e. point out errors without of- fering a better translation. In addition, there is no need for an efficient search algorithm, thus allowing for more complex models. 3. Target features judge the translation of the sys- tem without regarding in which way it was produced. They often resemble the language 91 model used in the noisy channel formulation (Brown et al., 1993) but can also pinpoint more specific issues. In practice, the same features …

Content truncated to 500 words