Feasibility of Human-in-the-loop Minimum Error Rate Training

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 52–61, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Feasibility of Human-in-the-loop Minimum Error Rate Training Omar F. Zaidan and Chris Callison-Burch Dept. of Computer Science, Johns Hopkins University Baltimore, MD 21218, USA {ozaidan,ccb}@cs.jhu.edu Abstract Minimum error rate training (MERT) in- volves choosing parameter values for a machine translation (MT) system that maximize performance on a tuning set as measured by an automatic evaluation met- ric, such as BLEU. The method is best when the system will eventually be eval- uated using the same metric, but in reality, most MT evaluations have a human-based component. Although performing MERT with a human-based metric seems like a daunting task, we describe a new metric, RYPT, which takes human judgments into account, but only requires human input to build a database that can be reused over and over again, hence eliminating the need for human input at tuning time. In this investigative study, we analyze the diver- sity (or lack thereof) of the candidates pro- duced during MERT, we describe how this redundancy can be used to our advantage, and show that RYPT is a better predictor of translation quality than BLEU. 1 Introduction Many state-of-the-art machine translation (MT) systems over the past few years (Och and Ney, 2002; Koehn et al., 2003; Chiang, 2007; Koehn et al., 2007; Li et al., 2009) rely on several mod- els to evaluate the “goodness” of a given candidate translation in the target language. The MT system proceeds by searching for the highest-scoring can- didate translation, as scored by the different model components, and returns that candidate as the hy- pothesis translation. Each of these models need not be a probabilistic model, and instead corre- sponds to a feature that is a function of a (can- didate translation,foreign sentence) pair. Treated as a log-linear model, we need to as- sign a weight for each of the features. Och (2003) shows that setting those weights should take into account the evaluation metric by which the MT system will eventually be judged. This is achieved by choosing the weights so as to maximize the per- formance of the MT system on a development set, as measured by that evaluation metric. The other insight of Och’s work is that there exists an ef- ficient algorithm to find such weights. This pro- cess has come to be known as the MERT phase (for Minimum Error Rate Training) in training pipelines of MT systems. A problem arises if the performance of the sys- tem is not judged by an automatic evaluation met- ric such as BLEU or TER, but instead through an evaluation process involving a human. The GALE evaluation, for instance, judges the quality of systems as measured by human-targeted TER (HTER), which computes the edit distance be- tween the system’s output and a version of the output post-edited by a human. The IWSLT and WMT workshops also have a manual evaluation component, as does the NIST …

Content truncated to 500 words