Review and analysis of China workshop on machine translation 2013 evaluation

Review and Analysis of China Workshop on Machine Translation 2013 Evaluation Sitong Yang1,2 yangsitong@ict.ac.cn Heng Yu1 yuheng@ict.ac.cn Hongmei Zhao1 zhaohongmei@ict.ac.cn Qun Liu1,3 qliu@computing.dcu.ie Yajuan l¨u1 lvyajuan@ict.ac.cn 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 CNGL, School of Computing, Dublin City University Abstract This paper gives a general review and detailed analysis of China Workshop on Machine Translation (CWMT) Evaluation. Compared with the past CWMT evaluation campaigns, CWMT2013 evaluation is characterized as follows: first, adopting gray-box evaluation which makes the results more replicable and controllable; second, adding one rule-based system as a counterpart; third, carrying out manual evaluations on some specific tasks to give a more com- prehensive analysis of the translation errors. Boosted by those new features, our analysis and case study on the evaluation results shows the pros and cons of both rule-based and statistical systems, and reveals some interesting correlations bewteen automatic and manual evaluation metrics on different translation systems. 1 Introduction The China Workshop on Machine Translation has always been focusing on catching the latest development of Machine Translation (MT) and promoting the communication between related organizations in China. By convention, we organized a unified machine translation evaluation in 2013, sponsored by Chinese Information Processing Society of China and held by the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS). Compared with the previous evaluation [Zhao et al., 2009], the main improvements of CWMT2013 are as follows: First, we follow the ”Gray-Box Evaluation” mode, which not only requires the participants to submit the final translation results but also Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 208 results of some key intermediate procedures as gray-box files, such as alignment results, k-best translation, etc. This mechanism makes the results more replicable and control- lable, at the same time it enables the participants to identify the weak link in their system pipeline, and make targeted adjustment to improve translation quality. Second, we adopt one rule-based system along with its statistical counterparts to make a more comprehensive comparison between different kinds of MT systems. To increase the diversity of evaluation method, two additional automatic evaluation metrics are also introduced: METEOR [Banerjee and Lavie, 2005] and TER [Snover et al., 2006]. Fi- nally, Besides the automatic evaluation, manual evaluation is also involved in this eval- uation. It provides not only the fidelity score and fluency score but also the error types of the translation. This will help us to identify the advantage for each system and the distribution of error types. Boosted by the above new features, our analysis and case study first shows that rule-based and statistical systems have very different error distributions, and the dis- tributions also vary with different domains. Second we find a serious discrepancy be- tween the automatic and manual evaluation results of the rule-based MT systems, and a detailed study in this problem reveals that automatic evaluation metrics such …

Content truncated to 500 words