Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 369–372, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators Omar F. Zaidan and Chris Callison-Burch Dept. of Computer Science, Johns Hopkins University Baltimore, MD 21218, USA {ozaidan,ccb}@cs.jhu.edu Abstract In the field of machine translation, automatic metrics have proven quite valuable in system development for tracking progress and meas- uring the impact of incremental changes. However, human judgment still plays a large role in the context of evaluating MT systems. For example, the GALE project uses human- targeted translation edit rate (HTER), wherein the MT output is scored against a post-edited version of itself (as opposed to being scored against an existing human reference). This poses a problem for MT researchers, since HTER is not an easy metric to calculate, and would require hiring and training human an- notators to perform the editing task. In this work, we explore soliciting those edits from untrained human annotators, via the online service Amazon Mechanical Turk. We show that the collected data allows us to predict HTER-ranking of documents at a significantly higher level than the ranking obtained using automatic metrics. 1 Introduction In the early days of machine translation (MT), it was typical to evaluate MT output by soliciting judgments from human subjects, such as evaluat- ing the fluency and adequacy of MT output (LDC, 2005). While this approach was appropriate (in- deed desired) for evaluating a system, it was not a practical means of tracking the progress of a sys- tem during its development, since collecting hu- man judgments is both costly and time-consuming. The introduction of automatic metrics like BLEU contributed greatly to MT research, for instance allowing researchers to measure and evaluate the impact of small modifications to an MT system. However, manual evaluation remains a core component of system evaluation. Teams on the GALE project, a DARPA-sponsored MT research program, are evaluated using the HTER metric, which is a version of TER whereby the output is scored against a post-edited version of itself, in- stead of a preexisting reference. Moreover, empha- sis is placed on performing well across all documents and across all genres. Therefore, it is important for a research team to be able to evaluate their system using HTER, or at least determine the ranking of the documents according to HTER, for purposes of error analysis. Instead of hiring a human translator and training them, we propose moving the task to the virtual world of Amazon’s Mechanical Turk (AMT), hiring workers to edit the MT output and predict HTER from those edits. We show that edits collected this way are better at predicting document ranking than automatic metrics, and furthermore that it can be done at a low cost, both in terms of time and money. The paper is organized as follows. We first discuss options available to predict HTER, such as automatic metrics. We then discuss the possibility of …

Content truncated to 500 words