CASICT-DCU Participation in WMT2015 Metrics Task

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 417–421, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. CASICT-DCU Participation in WMT2015 Metrics Task Hui Yu† Qingsong Ma† Xiaofeng Wu‡ Qun Liu‡† †Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences ‡ADAPT Centre, School of Computing, Dublin City University {yuhui,maqingsong}@ict.ac.cn {xiaofengwu}@computing.dcu.ie {liuqun}@ict.ac.cn Abstract Human-designed sub-structures are re- quired by most of the syntax-based ma- chine translation evaluation metrics. In this paper, we propose a novel evalua- tion metric based on dependency parsing model, which does not need this human in- volvement. Experimental results show that the new single metric gets better correla- tion than METEOR on system level and is comparable with it on sentence level. To introduce more information, we combine the new metric with many other metrics. The combined metric obtains state-of-the- art performance on both system level eval- uation and sentence level evaluation on WMT 2014. 1 Introduction Automatic evaluation metrics play an important role in machine translation research. At present, most of the automatic evaluation metrics evaluate the translation quality by comparing the similarity between the hypothesis and the reference. The lexicon-based metrics can only use lexical information, such as BLEU (Papineni et al., 2002), NIST(Doddington, 2002) and METEOR (Lavie and Agarwal, 2007). To evaluate the hypothe- sis on syntactic level, some researchers proposed the syntax-based metrics. Liu and Gildea (2005) proposed a constituent-tree-based metric STM and a dependency-tree-based metric HWCM. The syntax-based metric proposed by Owczarzak et al (2007) uses the Lexical-Functional Grammar (LFG) dependency tree. Some metrics introduce the syntactic information on the basis of lexi- cal information, such as MAXSIM (Chan and Ng, 2008) and the metric proposed by Zhu et al. (2010). These metrics evaluate the syntac- tic similarity by comparing the sub-structures ex- tracted from the trees of hypothesis and refer- ence. To avoid parsing the hypothesis in order to prevent translation error propagation, some re- searchers propose a kind of syntax-based evalua- tion metric which only uses the tree of reference, such as BLEU ˆATRE (Mehay and Brew, 2007) and RED (Yu et al., 2014). The syntax-based metrics either use the sub- structures of both the reference and the hypothesis tree, or only use that on the reference side. There- fore, for these metrics, sub-structures designed by human are required. In this paper, we propose a novel dependency-parsing-model-based metric in the view of dependency tree generation, which completely avoids this human involvement. A de- pendency parsing model is trained by the reference dependency tree, through which we can obtain the dependency tree of the hypothesis and the corre- sponding score. The syntactic similarity between the hypothesis and the reference can be evaluated by this score. In order to obtain the lexicon sim- ilarity, we also introduce the unigram F-score to the new metric. The experimental results show that the new metric gets the state-of-the-art per- formance in the single metrics on system level evaluation, and gets the comparable correlation with METEOR on …

Content truncated to 500 words