Quality Estimation: an experimental study using unsupervised similarity measures

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 120–126, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Quality Estimation: an experimental study using unsupervised similarity measures Erwan Moreau CNGL and Computational Linguistics Group Centre for Computing and Language Studies School of Computer Science and Statistics Trinity College Dublin Dublin 2, Ireland moreaue@cs.tcd.ie Carl Vogel Computational Linguistics Group Centre for Computing and Language Studies School of Computer Science and Statistics Trinity College Dublin Dublin 2, Ireland vogel@cs.tcd.ie Abstract We present the approach we took for our par- ticipation to the WMT12 Quality Estimation Shared Task: our main goal is to achieve rea- sonably good results without appeal to super- vised learning. We have used various simi- larity measures and also an external resource (Google N-grams). Details of results clarify the interest of such an approach. 1 Introduction Quality Estimation (or Confidence Estimation) refers here to the task of evaluating the quality of the output produced by a Machine Translation (MT) system. More precisely it consists in evaluating the quality of every individual sentence, in order (for in- stance) to decide whether a given sentence can be published as it is, should be post-edited, or is so bad that it should be manually re-translated. To our knowledge, most approaches so far (Spe- cia et al., 2009; Soricut and Echihabi, 2010; He et al., 2010; Specia et al., 2011) use several features combined together using supervised learning in or- der to predict quality scores. These features be- long to two categories: black box features which can be extracted given only the input sentence and its translated version, and glass box features which rely on various intermediate steps of the internal MT engine (thus require access to this internal data). For the features they studied, Specia et al. (2009) have shown that black box features are informative enough and glass box features do not significantly contribute to the accuracy of the predicted scores. In this study, we use only black box features, and further, eschew supervised learning except in the broadest sense. Our method requires some refer- ence data, all taken to be equally good exemplars of a positive reference category, against which the experimental sentences are compared automatically. This is the extent of broader-sense supervision. The method does not require a training set of items each annotated by human experts with quality scores (ex- cept for the purpose of evaluation of course). Successful unsupervised learning averts risks of the alternative: supervised learning necessarily makes the predicting system dependent on the an- notated training data, i.e. less generic, and requires a costly human evalution stage to obtain a reliable model. Of course, our approach is likely not to per- form as well as supervised approaches: here the goal is to find a rather generic robust way to measure quality, not to achieve the best accuracy. Neverthe- less, in the context of this Quality Evaluation Shared task (see (Callison-Burch et al., 2012) for a detailed description) we have also used supervised …

Content truncated to 500 words