A Graph-based Bilingual Corpus Selection Approach for SMT

A Graph-based Bilingual Corpus Selection Approach for SMT ∗ Wenhan Chao a and Zhoujun Li a a School of Computer Science & Engineering, BeiHang University, 37# Xueyuan Rd, Haidian District, Beijing 100191, China {chaowenhan, lizj}@buaa.edu.cn Abstract. In statistical machine translation, the number of sentence pairs in the bilingual corpus is very important to the quality of translation. However, when the quantity reaches some extent, enlarging the corpus has less effect on the translation quality; whereas increasing greatly the time and space complexity to train the translation model, which hinders the development of statistical machine translation. In this paper, we propose a graph-based bilingual corpus selection approach, which makes use of the structural information of corpus to measure and update the importance of each sentence pair, and then selects a sentence pair with the highest importance each time. Our experiments in a Chinese-English translation task show that, selecting only 50% of the whole corpus by the graph-based selection approach as training set, we can obtain the near translation result with the one using the whole corpus. Keywords: Statistical Machine Translation, Corpus Selection, Graph ∗ The work reported in this paper was supported by National Natural Science Foundation of China, Contract No. 61003111; and supported by Research Fund for the Doctoral Program of Higher Education of China (New teacher Fund), Contract No. 20101102120016 Copyright 2011 by WenHan Chao, ZhouJun Li 1 Introduction In statistical machine translation, large scale of bilingual corpus is very important. In order to improve the quality of translation, there are two viewpoints about the use of corpus. One way is to collect more and more bilingual corpus to improve the quality of translation model, such as extracting the sentence pairs from the comparable corpus (Smith et al., 2010; Uszkoreit et al., 2010). However, some researchers found that, after the quantity of the sentence pairs (Han et al., 2009) in the corpus reaches some extent, adding more sentence pairs will not improve the quality of translation significantly. On the other hand, larger and larger corpus will consume more and more resources, which hinders the research progress of machine translation in some degree. This type of approaches assumes the sentence pairs in the corpus are independent each other, not considering the relationship between sentence pairs and their effect on the translation. The other view is to mine the potential of training corpus through corpus selection and optimization to improve the quality of the translation model. And it also includes three ways: the first one is to select and optimize the training corpus to adapt to the test set (Lu et al., 2007) or the domain (Yasuda et al,. 2008); the second one is to select the sentence pairs with high quality as training corpus (Chen et al., 2006; Han et al., 2009), in which the quality is measured through the features of the sentence pair itself, such as the number of words that can be translated each other in the sentence pair; the third one is to measure and sort …

Content truncated to 500 words