Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 403–411, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment Jason R. Smith∗ Center for Lang. and Speech Processing Johns Hopkins University Baltimore, MD 21218 jsmith@cs.jhu.edu Chris Quirk and Kristina Toutanova Microsoft Research One Microsoft Way Redmond, WA 98052 {chrisq,kristout}@microsoft.com Abstract The quality of a statistical machine transla- tion (SMT) system is heavily dependent upon the amount of parallel sentences used in train- ing. In recent years, there have been several approaches developed for obtaining parallel sentences from non-parallel, or comparable data, such as news articles published within the same time period (Munteanu and Marcu, 2005), or web pages with a similar structure (Resnik and Smith, 2003). One resource not yet thoroughly explored is Wikipedia, an on- line encyclopedia containing linked articles in many languages. We advance the state of the art in parallel sentence extraction by modeling the document level alignment, mo- tivated by the observation that parallel sen- tence pairs are often found in close proximity. We also include features which make use of the additional annotation given by Wikipedia, and features using an automatically induced lexicon model. Results for both accuracy in sentence extraction and downstream im- provement in an SMT system are presented. 1 Introduction For any statistical machine translation system, the size of the parallel corpus used for training is a ma- jor factor in its performance. For some language pairs, such as Chinese-English and Arabic-English, large amounts of parallel data are readily available, but for most language pairs this is not the case. The ∗This research was conducted during the author’s intern- ship at Microsoft Research. domain of the parallel corpus also strongly influ- ences the quality of translations produced. Many parallel corpora are taken from the news domain, or from parliamentary proceedings. Translation qual- ity suffers when a system is not trained on any data from the domain it is tested on. While parallel corpora may be scarce, compara- ble, or semi-parallel corpora are readily available in several domains and language pairs. These cor- pora consist of a set of documents in two languages containing similar information. (See Section 2.1 for a more detailed description of the types of non- parallel corpora.) In most previous work on ex- traction of parallel sentences from comparable cor- pora, some coarse document-level similarity is used to determine which document pairs contain paral- lel sentences. For identifying similar web pages, Resnik and Smith (2003) compare the HTML struc- ture. Munteanu and Marcu (2005) use publication date and vector-based similarity (after projecting words through a bilingual dictionary) to identify similar news articles. Once promising document pairs are identified, the next step is to extract parallel sentences. Usu- ally, some seed parallel data is assumed to be avail- able. This data is used to train a word align- ment model, such as IBM Model 1 (Brown et al., 1993) or HMM-based word …

Content truncated to 500 words