A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 880–887, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation Joshua S. Albrecht and Rebecca Hwa Department of Computer Science University of Pittsburgh {jsa8,hwa}@cs.pitt.edu Abstract Recent studies suggest that machine learn- ing can be applied to develop good auto- matic evaluation metrics for machine trans- lated sentences. This paper further ana- lyzes aspects of learning that impact per- formance. We argue that previously pro- posed approaches of training a Human- Likeness classifier is not as well correlated with human judgments of translation qual- ity, but that regression-based learning pro- duces more reliable metrics. We demon- strate the feasibility of regression-based metrics through empirical analysis of learn- ing curves and generalization studies and show that they can achieve higher correla- tions with human judgments than standard automatic metrics. 1 Introduction As machine translation (MT) research advances, the importance of its evaluation also grows. Efficient evaluation methodologies are needed both for facili- tating the system development cycle and for provid- ing an unbiased comparison between systems. To this end, a number of automatic evaluation metrics have been proposed to approximate human judg- ments of MT output quality. Although studies have shown them to correlate with human judgments at the document level, they are not sensitive enough to provide reliable evaluations at the sentence level (Blatz et al., 2003). This suggests that current met- rics do not fully reflect the set of criteria that people use in judging sentential translation quality. A recent direction in the development of met- rics for sentence-level evaluation is to apply ma- chine learning to create an improved composite met- ric out of less indicative ones (Corston-Oliver et al., 2001; Kulesza and Shieber, 2004). Under the as- sumption that good machine translation will pro- duce “human-like” sentences, classifiers are trained to predict whether a sentence is authored by a human or by a machine based on features of that sentence, which may be the sentence’s scores from individ- ual automatic evaluation metrics. The confidence of the classifier’s prediction can then be interpreted as a judgment on the translation quality of the sentence. Thus, the composite metric is encoded in the confi- dence scores of the classification labels. While the learning approach to metric design of- fers the promise of ease of combining multiple met- rics and the potential for improved performance, several salient questions should be addressed more fully. First, is learning a “Human Likeness” classi- fier the most suitable approach for framing the MT- evaluation question? An alternative is regression, in which the composite metric is explicitly learned as a function that approximates humans’ quantitative judgments, based on a set of human evaluated train- ing sentences. Although regression has been con- sidered on a small scale for a single system as con- fidence estimation (Quirk, 2004), this approach has not been studied as extensively due to scalability and generalization concerns. Second, how does …

Content truncated to 500 words