Measurement of Progress in Machine Translation

Measurement of Progress in Machine Translation Yvette Graham Timothy Baldwin Aaron Harwood Alistair Moffat Justin Zobel Department of Computing and Information Systems The University of Melbourne {ygraham,tbaldwin,aharwood,amoffat,jzobel}@unimelb.edu.au Abstract Machine translation (MT) systems can only be improved if their performance can be reli- ably measured and compared. However, mea- surement of the quality of MT output is not straightforward, and, as we discuss in this paper, relies on correlation with inconsistent human judgments. Even when the question is captured via “is translation A better than translation B” pairwise comparisons, empiri- cal evidence shows that inter-annotator consis- tency in such experiments is not particularly high; for intra-judge consistency – computed by showing the same judge the same pair of candidate translations twice – only low levels of agreement are achieved. In this paper we review current and past methodologies for hu- man evaluation of translation quality, and ex- plore the ramifications of current practices for automatic MT evaluation. Our goal is to docu- ment how the methodologies used for collect- ing human judgments of machine translation quality have evolved; as a result, we raise key questions in connection with the low levels of judgment agreement and the lack of mecha- nisms for longitudinal evaluation. 1 Introduction Measurement is central to all scientific endeavor. In computing, we rely on impartial and scrutable eval- uations of phenomena in order to determine the ex- tent to which progress is being made in that disci- pline area. We then use those measurements to pre- dict performance on unseen data. That is, we need accurate measurement to know that we have made progress, and we need those measurements to be predictive, so that we can have confidence that we will benefit from the improvements that have been attained. The particular focus of this paper is mea- surement of translation quality in the field of ma- chine translation (MT). In some areas of computing, measurement tech- niques are unambiguous, directly comparable be- tween systems, and enduring over time. For exam- ple, a proposed new approach to text compression can be evaluated on a wide range of files, and the criteria to be measured in each case are straightfor- ward: execution time for encoding and decoding; memory space used during encoding and decoding; and, of course, compressed file size. All of these facets are objective, in that, if the same file is com- pressed a second time on the same hardware, the same measurements (to within some predictable tol- erance, in the case of execution speed) will result; and compressing the same file with the same tech- nique on different hardware ten years later should still result in consistent measures of memory use and file size. To compare two approaches to text com- pression, therefore, the only real complexity is in assembling a collection of documents which is “rep- resentative” of utility in general or over some spe- cific domain (for example, compression of micro- posts from a service such as Twitter). Beyond this, as long as the evaluation …

Content truncated to 500 words