An Empirical Study on Development Set Selection Strategy for Machine Translation Learning

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 67–71, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics An Empirical Study on Development Set Selection Strategy for Machine Translation Learning∗ Cong Hui12, Hai Zhao12†, Yan Song3, Bao-Liang Lu12 1Center for Brain-Like Computing and Machine Intelligence Department of Computer Science and Engineering, Shanghai Jiao Tong University 2MOE-Microsoft Key Laboratory for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong University, 800 Dong Chuan Rd., Shanghai 200240, China 3Department of Chinese, Translation and Linguistics, City University of Hong Kong huicong@sjtu.edu.cn, {zhaohai,blu}@cs.sjtu.edu.cn Abstract This paper describes a statistical machine translation system for our participation for the WMT10 shared task. Based on MOSES, our system is capable of translat- ing German, French and Spanish into En- glish. Our main contribution in this work is about effective parameter tuning. We discover that there is a significant perfor- mance gap as different development sets are adopted. Finally, ten groups of devel- opment sets are used to optimize the model weights, and this does help us obtain a sta- ble evaluation result. 1 Introduction We present a machine translation system that rep- resents our participation for the WMT10 shared task from Brain-like Computing and Machine In- telligence Lab of Shanghai Jiao Tong University (SJTU-BCMI Lab). The system is based on the state-of-the-art SMT toolkit MOSES (Koehn et al., 2007). We use it to translate German, French and Spanish into English. Though different develop- ment sets used for training parameter tuning will certainly lead to quite different performance, we empirically find that the more sets we combine to- gether, the more stable the performance is, and a development set similar with test set will help the performance improvement. 2 System Description The basic model of the our system is a log-linear model (Och and Ney, 2002). For given source lan- ∗This work was partially supported by the National Natu- ral Science Foundation of China (Grant No. 60903119, Grant No. 60773090 and Grant No. 90820018), the National Basic Research Program of China (Grant No. 2009CB320901), and the National High-Tech Research Program of China (Grant No.2008AA02Z315). †corresponding author guage strings, the target language string t will be obtained by the following equation, ˆtI 1 = arg max tI 1 {pλm 1 (tI 1 | sJ 1 )} = arg max tI 1 { exp[�M m=1 λmhm(tI 1, sJ 1 )] � ¯tI 1 exp[�M m=1 λmhm(¯tI 1, sJ 1 )] }, where hm is the m-th feature function and λm is the m-th model weight. There are four main parts of features in the model: translation model, lan- guage model, reordering model and word penalty. The whole model has been well implemented by the state-of-the-art statistical machine translation toolkit MOSES. For each language that is required to translated into English, two sets of bilingual corpora are pro- vided by the shared task organizer. The first set is the new release (version 5) of Europarl cor- pus which is the smaller. The second is a com- …

Content truncated to 500 words