Source-Language Features and Maximum Correlation Training for Machine Translation Evaluation

Proceedings of NAACL HLT 2007, pages 41–48, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Source-Language Features and Maximum Correlation Training for Machine Translation Evaluation Ding Liu and Daniel Gildea Department of Computer Science University of Rochester Rochester, NY 14627 Abstract We propose three new features for MT evaluation: source-sentence constrained n-gram precision, source-sentence re- ordering metrics, and discriminative un- igram precision, as well as a method of learning linear feature weights to directly maximize correlation with human judg- ments. By aligning both the hypothe- sis and the reference with the source- language sentence, we achieve better cor- relation with human judgments than pre- viously proposed metrics. We further improve performance by combining indi- vidual evaluation metrics using maximum correlation training, which is shown to be better than the classification-based frame- work. 1 Introduction Evaluation has long been a stumbling block in the development of machine translation systems, due to the simple fact that there are many correct trans- lations for a given sentence. The most commonly used metric, BLEU, correlates well over large test sets with human judgments (Papineni et al., 2002), but does not perform as well on sentence-level eval- uation (Blatz et al., 2003). Later approaches to im- prove sentence-level evaluation performance can be summarized as falling into four types: • Metrics based on common loose sequences of MT outputs and references (Lin and Och, 2004; Liu and Gildea, 2006). Such metrics were shown to have better fluency evaluation per- formance than metrics based on n-grams such BLEU and NIST (Doddington, 2002). • Metrics based on syntactic similarities such as the head-word chain metric (HWCM) (Liu and Gildea, 2005). Such metrics try to improve flu- ency evaluation performance for MT, but they heavily depend on automatic parsers, which are designed for well-formed sentences and cannot generate robust parse trees for MT outputs. • Metrics based on word alignment between MT outputs and the references (Banerjee and Lavie, 2005). Such metrics do well in adequacy evalu- ation, but are not as good in fluency evaluation, because of their unigram basis (Liu and Gildea, 2006). • Combination of metrics based on machine learning. Kulesza and Shieber (2004) used SVMs to combine several metrics. Their method is based on the assumption that higher classification accuracy in discriminat- ing human- from machine-generated transla- tions will yield closer correlation with human judgment. This assumption may not always hold, particularly when classification is diffi- cult. Lita et al. (2005) proposed a log-linear model to combine features, but they only did preliminary experiments based on 2 features. Following the track of previous work, to improve evaluation performance, one could either propose new metrics, or find more effective ways to combine the metrics. We explore both approaches. Much work has been done on computing MT scores based 41 on the pair of MT output/reference, and we aim to investigate whether some other information could be used in the MT evaluation, such as source sen- tences. We propose two types of source-sentence related features as …

Content truncated to 500 words