Is Machine Translation Getting Better over Time?

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Is Machine Translation Getting Better over Time? Yvette Graham Timothy Baldwin Alistair Moffat Department of Computing and Information SystemsJustin Zobel The University of Melbourne {ygraham,tbaldwin,ammoffat,jzobel}@unimelb.edu.au Abstract Recent human evaluation of machine translation has focused on relative pref- erence judgments of translation quality, making it difficult to track longitudinal im- provements over time. We carry out a large-scale crowd-sourcing experiment to estimate the degree to which state-of-the- art performance in machine translation has increased over the past five years. To fa- cilitate longitudinal evaluation, we move away from relative preference judgments and instead ask human judges to provide direct estimates of the quality of individ- ual translations in isolation from alternate outputs. For seven European language pairs, our evaluation estimates an aver- age 10-point improvement to state-of-the- art machine translation between 2007 and 2012, with Czech-to-English translation standing out as the language pair achiev- ing most substantial gains. Our method of human evaluation offers an economi- cally feasible and robust means of per- forming ongoing longitudinal evaluation of machine translation. 1 Introduction Human evaluation provides the foundation for em- pirical machine translation (MT), whether human judges are employed directly to evaluate system output, or via the use of automatic metrics – validated through correlation with human judg- ments. Achieving consistent human evaluation is not easy, however. Annual evaluation cam- paigns conduct large-scale human assessment but report ever-decreasing levels of judge consistency – when given the same pair of translations to repeat-assess, even expert human judges will wor- ryingly often contradict both the preference judg- ment of other judges and their own earlier prefer- ence (Bojar et al., 2013). For this reason, human evaluation has been targeted within the commu- nity as an area in need of attention, with increased efforts to develop more reliable methodologies. One standard platform for human evaluation is WMT shared tasks, where assessments have (since 2007) taken the form of ranking five alternate sys- tem outputs from best to worst (Bojar et al., 2013). This method has been shown to produce more con- sistent judgments compared to fluency and ade- quacy judgments on a five-point scale (Callison- Burch et al., 2007). However, relative preference judgments have been criticized for being a sim- plification of the real differences between trans- lations, not sufficiently taking into account the large number of different types of errors of vary- ing severity that occur in translations (Birch et al., 2013). Relative preference judgments do not take into account the degree to which one translation is better than another – there is no way of knowing if a winning system produces far better translations than all other systems, or if that system would have ranked lower if the severity of its inferior transla- tion outputs were taken into account. Rather than directly aiming to increase human judge consistency, some methods instead increase the number of reference …

Content truncated to 500 words