RED, The DCU-CASICT Submission of Metrics Tasks

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 420–425, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics RED: DCU-CASICT Participation in WMT2014 Metrics Task Xiaofeng Wu†, Hui Yu∗, Qun Liu†∗ †CNGL Centre for Global Intelligent Content School of Computing, Dublin City University Dublin 9, Ireland ∗Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences Beijing, China {xiaofengwu,qliu}@computing.dcu.ie, yuhui@ict.ac.cn Abstract Based on the last year’s DCU-CASIST participation on WMT metrics task, we further improve our model in the follow- ing ways: 1) parameter tuning 2) support languages other than English. We tuned our system on all the data of WMT 2010, 2012 and 2013. The tuning results as well as the WMT 2014 test results are reported. 1 Introduction Automatic evaluation plays a more and more im- portant role in the evolution of machine transla- tion. There are roughly two categories can be seen: namely lexical information based and structural information based. 1) Lexical information based approaches, among which, BLEU (?), Translation Error Rate (TER) (?) and METEOR (?) are the most popular ones and, with simplicity as their merits, cannot adequately reflect the structural level similarity. 2) A lot of structural level based methods have been exploited to overcome the weakness of the lexical based methods, including Syntactic Tree Model(STM)(?), a constituent tree based ap- proach, and Head Word Chain Model(HWCM)(?), a dependency tree based approach. Both of the methods compute the similarity between the sub-trees of the hypothesis and the reference. Owczarzak et al (?; ?; ?) presented a method using the Lexical-Functional Grammar (LFG) de- pendency tree. MAXSIM (?) and the method pro- posed by Zhu et al (?) also employed the syntac- tic information in association with lexical infor- mation. As we know that the hypothesis is poten- tially noisy, and these errors are enlarged through the parsing process. Thus the power of syntactic information could be considerably weakened. In this paper, based on our attempt on WMT metrics task 2013 (?), we propose a metrics named RED ( REference Dependency based automatic evaluation method). Our metrics employs only the reference dependency tree which contains both the lexical and syntactic information, leaving the hy- pothesis side unparsed to avoid error propagation. 2 Parameter Tuning In RED, we use F-score as our final score. F-score is calculated by Formula (1), where α is a value between 0 and 1. F-score = precision · recall α · precision + (1 − α) · recall (1) The dependency tree of the reference and the string of the translation are used to calculate the precision and recall. In order to calculate preci- sion, the number of the dep-ngrams (the ngrams obtained from dependency tree1) should be given, but there is no dependency tree for the transla- tion in our method. We know that the number of dep-ngrams has an approximate linear relation- ship with the length of the sentence, so we use the length of the translation to replace the number …

Content truncated to 500 words