Phrase-based Parallel Fragments Extraction from Comparable Corpora

International Joint Conference on Natural Language Processing, pages 972–976, Nagoya, Japan, 14-18 October 2013. Phrase-based Parallel Fragments Extraction from Comparable Corpora Xiaoyin Fu, Wei Wei, Shixiang Lu, Zhenbiao Chen and Bo Xu Interactive Digital Media Technology Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China {xiaoyin.fu,wei.wei,shixiang.lu,zhenbiao.chen,xubo}@ia.ac.cn Abstract We present a phrase-based method to ex- tract parallel fragments from the compa- rable corpora. We do this by introducing a force decoder based on the hierarchical phrase-based (HPB) translation model to detect the alignments in comparable sen- tence pairs. This method enables us to ex- tract useful training data for statistical ma- chine translation (SMT) system. We eval- uate our method by fragment detection and large-scale translation tasks, which show that our method can effectively extract par- allel fragments and improve the perfor- mance of the state-of-the-art SMT system. 1 Introduction Parallel corpora are valuable resources for train- ing a statistical translation system. In most cases, it has been an effective way to build state-of-the- art statistical models using a large scale of paral- lel corpora. However, the parallel corpora only exist in particular domains for a few number of language pairs, such as international conference recordings and legal texts. Since comparable cor- pora exist in large quantities with many languages, and the exploitation in them for extracting parallel data can be very useful for SMT system, the ac- quisition of parallel data from comparable corpora has caught much attention. Various methods (Zhao and Vogel, 2002; Munteanu and Marcu, 2005; Abdul-Rauf and Schwenk, 2009; Smith et al., 2010) have been previously proposed to extract parallel data from comparable corpora at the sentence level. These methods share the same framework, which firstly identifies candidate document pairs and then ex- tracts parallel sentences from the obtained docu- ments. However, it is found that most of these sentences are comparable sentence pairs (Hong et al., 2010), which embed non-parallel fragments or even lack translations. Consider the comparable sentence pair from Chinese to English in Figure 1. Methods for extracting parallel sentences will bring in noise when these bilingual sentences are retained. But discarding them is also not a wise choice, as there are still some useful parallel frag- ments as the underlines shown in the figure. Figure 1: Example of comparable sentence pairs. The parallel fragments are marked by underlines. In order to deal with this problem, further effort- s (Munteanu and Marcu, 2006; Quirk et al., 2007; Kumano et al., 2007; Lardilleux et al., 2012) were made to obtain parallel data at the fragment lev- el. The work of (Riesa and Marcu, 2012) detect- ed parallel fragments using the hierarchical align- ment model. However, this approach obtains frag- ments from parallel sentence pairs, which limits it- s application in comparable corpora. (Hewavitha- rana and Vogel, 2011) have explored several align- ment approaches to detect parallel fragments em- bedded in comparable sentences. However, these approaches extract fragments mainly using the lexical features and considering the words in par- allel fragments are independent, which …

Content truncated to 500 words