A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 414–421, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics A Description of Tunable Machine Translation Evaluation Systems in WMT13 Metrics Task Aaron L.-F. Han hanlifengaaron@gmail.com Derek F. Wong derekfw@umac.mo Lidia S. Chao lidiasc@umac.mo Yi Lu mb25435@umac.mo Liangye He wutianshui0515@gmail.com Yiming Wang mb25433@umac.mo Jiaji Zhou mb25473@uamc.mo Natural Language Processing & Portuguese-Chinese Machine Translation Laboratory Department of Computer and Information Science University of Macau, Macau S.A.R. China Abstract This paper is to describe our machine transla- tion evaluation systems used for participation in the WMT13 shared Metrics Task. In the Metrics task, we submitted two automatic MT evaluation systems nLEPOR_baseline and LEPOR_v3.1. nLEPOR_baseline is an n-gram based language independent MT evaluation metric employing the factors of modified sen- tence length penalty, position difference penal- ty, n-gram precision and n-gram recall. nLEPOR_baseline measures the similarity of the system output translations and the refer- ence translations only on word sequences. LEPOR_v3.1 is a new version of LEPOR met- ric using the mathematical harmonic mean to group the factors and employing some linguis- tic features, such as the part-of-speech infor- mation. The evaluation results of WMT13 show LEPOR_v3.1 yields the highest average- score 0.86 with human judgments at system- level using Pearson correlation criterion on English-to-other (FR, DE, ES, CS, RU) lan- guage pairs. 1 Introduction Machine translation has a long history since the 1950s (Weaver, 1955) and gains a fast develop- ment in the recent years because of the higher level of computer technology. For instances, Och (2003) presents Minimum Error Rate Training (MERT) method for log-linear statistical ma- chine translation models to achieve better trans- lation quality; Menezes et al. (2006) introduce a syntactically informed phrasal SMT system for English-to-Spanish translation using a phrase translation model, which is based on global reor- dering and the dependency tree; Su et al. (2009) use the Thematic Role Templates model to im- prove the translation; Costa-jussà et al. (2012) develop the phrase-based SMT system for Chi- nese-Spanish translation using a pivot language. With the rapid development of Machine Transla- tion (MT), the evaluation of MT has become a challenge in front of researchers. However, the MT evaluation is not an easy task due to the fact of the diversity of the languages, especially for the evaluation between distant languages (Eng- lish, Russia, Japanese, etc.). 2 Related works The earliest human assessment methods for ma- chine translation include the intelligibility and fidelity used around 1960s (Carroll, 1966), and the adequacy (similar as fidelity), fluency and comprehension (improved intelligibility) (White et al., 1994). Because of the expensive cost of manual evaluations, the automatic evaluation metrics and systems appear recently. The early automatic evaluation metrics in- clude the word error rate WER (Su et al., 1992) and position independent word error rate PER (Tillmann et al., 1997) that are based on the Le- venshtein distance. Several promotions for the MT and MT evaluation literatures include the ACL’s annual workshop on statistical machine translation WMT (Koehn and Monz, …

Content truncated to 500 words