Estimating the Sentence-Level Quality of Machine Translation Systems

Proceedings of the 13th Annual Conference of the EAMT, pages 28–35, Barcelona, May 2009 Estimating the Sentence-Level Quality of Machine Translation Systems Lucia Specia*, Nicola Cancedda and Marc Dymetman Xerox Research Centre Europe Meylan, 38240, France lucia.specia@xrce.xerox.com nicola.cancedda@xerox.com marc.dymetman@xerox.com Marco Turchi* and Nello Cristianini Department of Engineering Mathematics University of Bristol Bristol, BS8 1TR, UK Marco.Turchi@bristol.ac.uk nello@support-vector.net Abstract We investigate the problem of predicting the quality of sentences produced by ma- chine translation systems when reference translations are not available. The prob- lem is addressed as a regression task and a method that takes into account the con- tribution of different features is proposed. We experiment with this method for trans- lations produced by various MT systems and different language pairs, annotated with quality scores both automatically and manually. Results show that our method allows obtaining good estimates and that identifying a reduced set of relevant fea- tures plays an important role. The experi- ments also highlight a number of outstand- ing features that were consistently selected as the most relevant and could be used in different ways to improve MT perfor- mance or to enhance MT evaluation. 1 Introduction The notion of “quality” in Machine Translation (MT) can have different interpretations depend- ing on the intended use of the translations (e.g., fluency and adequacy, post-editing time, etc.). Nonetheless, the assessment of the quality of a translation is in general done by the user, who needs to read the translation and the source text to be able to judge whether it is a good transla- tion or not. This is a very time consuming task and may not even be possible, if the user does not have knowledge about the source language. There- fore, automatically assessing the quality of trans- *L. Specia and M. Turchi contributed equally to this work. c⃝ 2009 European Association for Machine Translation. lations produced by MT systems is a crucial prob- lem, either to filter out the low quality ones, e.g. to avoid professional translators spending time read- ing / post-editing bad translations, or to present them in such a way as to make end-users aware of the quality. This task, referred to as Confidence Estimation (CE), is concerned about predicting the quality of a system’s output for a given input, with- out any information about the expected output. CE for MT has been viewed as a binary classi- fication problem (Blatz et al., 2003) to distinguish between “good” and “bad” translations. However, it may be difficult to find a clear boundary between “good” and “bad” translations and this information may not be useful in certain applications (e.g, the time necessary to post-edit translations). We distinguish the task of CE from that of MT evaluation by the need, in the latter, of reference translations. The general goal of MT evaluation is to compare a machine translation to reference translation(s) and provide a quality score which re- flects how close the two translations are. In CE, the task consists in estimating the quality of the trans- …

Content truncated to 500 words