E-rating Machine Translation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 108–115, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics E-rating Machine Translation Kristen Parton1 Joel Tetreault2 Nitin Madnani2 Martin Chodorow3 1Columbia University, NY, USA kristen@cs.columbia.edu 2Educational Testing Service, Princeton, NJ, USA {jtetreault, nmadnani}@ets.org 3Hunter College of CUNY, New York, NY, USA martin.chodorow@hunter.cuny.edu Abstract We describe our submissions to the WMT11 shared MT evaluation task: MTeRater and MTeRater-Plus. Both are machine-learned metrics that use features from e-rater R⃝, an au- tomated essay scoring engine designed to as- sess writing proficiency. Despite using only features from e-rater and without comparing to translations, MTeRater achieves a sentence- level correlation with human rankings equiva- lent to BLEU. Since MTeRater only assesses fluency, we build a meta-metric, MTeRater- Plus, that incorporates adequacy by combin- ing MTeRater with other MT evaluation met- rics and heuristics. This meta-metric has a higher correlation with human rankings than either MTeRater or individual MT metrics alone. However, we also find that e-rater fea- tures may not have significant impact on cor- relation in every case. 1 Introduction The evaluation of machine translation (MT) systems has received significant interest over the last decade primarily because of the concurrent rising interest in statistical machine translation. The majority of re- search on evaluating translation quality has focused on metrics that compare translation hypotheses to a set of human-authored reference translations. How- ever, there has also been some work on methods that are not dependent on human-authored translations. One subset of such methods is task-based in that the methods determine the quality of a translation in terms of how well it serves the need of an extrin- sic task. These tasks can either be downstream NLP tasks such as information extraction (Parton et al., 2009) and information retrieval (Fujii et al., 2009) or human tasks such as answering questions on a read- ing comprehension test (Jones et al., 2007). Besides extrinsic evaluation, there is another set of methods that attempt to “learn” what makes a good translation and then predict the quality of new translations without comparing to reference trans- lations. Corston-Oliver et al. (2001) proposed the idea of building a decision tree classifier to sim- ply distinguish between machine and human transla- tions using language model (LM) and syntactic fea- tures. Kulesza and Shieber (2004) attempt the same task using an support vector machine (SVM) classi- fier and features derived from reference-based MT metrics such as WER, PER, BLEU and NIST. They also claim that the confidence score for the classi- fier being used, if available, may be taken as an es- timate of translation quality. Quirk (2004) took a different approach and examined whether it is pos- sible to explicitly compute a confidence measure for each translated sentence by using features derived from both the source and target language sides. Al- brecht and Hwa (2007a) expanded on this idea and conducted a larger scale study to show the viabil- ity of regression as a sentence-level metric of MT quality. …

Content truncated to 500 words