DCU Participation in WMT2013 Metrics Task

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 435–439, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics DCU Participation in WMT2013 Metrics Task Xiaofeng Wu†, Hui Yu⋆,Qun Liu† †CNGL, Dublin City University, Ireland ⋆ICT, Chinese Academy of Sciences, China †{xfwu, qliu}@computing.dcu.ie ⋆yuhui@ict.ac.cn Abstract In this paper, we propose a novel syntac- tic based MT evaluation metric which only employs the dependency information in the source side. Experimental results show that our method achieves higher correla- tion with human judgments than BLEU, TER, HWCM and METEOR at both sen- tence and system level for all of the four language pairs in WMT 2010. 1 Introduction Automatic evaluation plays a more important role in the evolution of machine translation. At the ear- liest stage, the automatic evaluation metrics only use the lexical information, in which, BLEU (Pap- ineni et al., 2002) is the most popular one. BLEU is simple and effective. Most of the researchers regard BLEU as their primary evaluation metric to develop and compare MT systems. However, BLEU only employs the lexical information and cannot adequately reflect the structural level sim- ilarity. Translation Error Rate (TER) (Snover et al., 2006) measures the number of edits required to change the hypothesis into one of the references. METEOR (Lavie and Agarwal, 2007), which de- fines loose unigram matching between the hypoth- esis and the references with the help of stem- ming and Wordnet-looking-up, is also a lexical based method and achieves the first-class human- evaluation-correlation score. AMBER (Chen and Kuhn, 2011; Chen et al., 2012) incorporates recall, extra penalties and some text processing variants on the basis of BLEU. The main weakness of all the above lexical based methods is that they cannot adequately reflect the structural level similarity. To overcome the weakness of the lexical based methods, many syntactic based metrics were pro- posed. Liu and Gildea (2005) proposed STM, a constituent tree based approach, and HWCM, a dependency tree based approach. Both of the two methods compute the similar- ity between the sub-trees of the hypothesis and the reference. Owczarzak et al (2007a; 2007b; 2007c) presented a method using the Lexical-Functional Grammar (LFG) dependency tree. MAXSIM (Chan and Ng, 2008) and the method proposed by Zhu et al (2010) also employed the syntac- tic information in association with lexical infor- mation.With the syntactic information which can reflect structural information, the correlation with the human judgments can be improved to a certain extent. As we know that the hypothesis is potentially noisy, and these errors expand through the parsing process. Thus the power of syntactic information could be considerably weakened. In this paper, we attempt to overcome the short- coming of the syntactic based methods and pro- pose a novel dependency based MT evaluation metric. The proposed metric only employs the ref- erence dependency tree which contains both the lexical and syntactic information, leaving the hy- pothesis side unparsed to avoid the error propaga- tion. In our metric, F-score is calculated using the string of hypothesis and …

Content truncated to 500 words