Accurate Parallel Fragment Extraction from Quasi–Comparable Corpora using Alignment Model and Translation Lexicon

International Joint Conference on Natural Language Processing, pages 1144–1150, Nagoya, Japan, 14-18 October 2013. Accurate Parallel Fragment Extraction from Quasi–Comparable Corpora using Alignment Model and Translation Lexicon Chenhui Chu, Toshiaki Nakazawa, Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku Kyoto, 606-8501, Japan {chu,nakazawa}@nlp.ist.i.kyoto-u.ac.jp kuro@i.kyoto-u.ac.jp Abstract Although parallel sentences rarely exist in quasi–comparable corpora, there could be parallel fragments that are also helpful for statistical machine translation (SMT). Previous studies cannot accurately extract parallel fragments from quasi–comparable corpora. To solve this problem, we pro- pose an accurate parallel fragment extrac- tion system that uses an alignment model to locate the parallel fragment candidates, and uses an accurate lexicon filter to iden- tify the truly parallel ones. Experimen- tal results indicate that our system can accurately extract parallel fragments, and our proposed method significantly outper- forms a state–of–the–art approach. Fur- thermore, we investigate the factors that may affect the performance of our system in detail. 1 Introduction In statistical machine translation (SMT) (Brown et al., 1993; Koehn et al., 2007), since trans- lation knowledge is acquired from parallel data, the quality and quantity of parallel data are crucial. However, except for a few language pairs, such as English–French, English–Arabic, English–Chinese and several European language pairs, parallel data remains a scarce resource. As non–parallel corpora are far more available, ex- tracting parallel data from non–parallel corpora is an attractive research field. Most previous studies focus on extracting par- allel sentences from comparable corpora (Zhao and Vogel, 2002; Utiyama and Isahara, 2003; Munteanu and Marcu, 2005; Tillmann, 2009; Smith et al., 2010; Abdul-Rauf and Schwenk, 2011). Quasi–comparable corpora that contain far more disparate very–non–parallel bilingual docu- ments that could either be on the same topic (in– topic) or not (out–topic) (Fung and Cheung, 2004), are available in far larger quantities than compara- ble corpora. In quasi–comparable corpora, there are few or no parallel sentences. However, there could be parallel fragments in comparable sen- tences that are also helpful for SMT. Previous studies for parallel fragment extraction from comparable sentences have the problem that they cannot extract parallel fragments accurately. Some studies extract parallel fragments relying on a probabilistic translation lexicon estimated on an external parallel corpus. They locate the source and target fragments independently, mak- ing the extracted fragments unreliable (Munteanu and Marcu, 2006). Some studies develop align- ment models for comparable sentences to extract parallel fragments (Quirk et al., 2007). Because the comparable sentences are quite noisy, the ex- tracted fragments are not accurate. In this paper, we propose an accurate parallel fragment extraction system. We locate parallel fragment candidates using an alignment model, and use an accurate lexicon filter to identify the truly parallel ones. Experimental results on Chinese–Japanese corpora show that our proposed method significantly outperforms a state–of–the– art approach, which indicate the effectiveness of our parallel fragment extraction system. More- over, we investigate the factors that may affect the performance of our system in detail. 2 Related Work (Munteanu and Marcu, 2006) is the first attempt to extract …

Content truncated to 500 words