An Empirical Study on Web Mining of Parallel Data

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 474–482, Beijing, August 2010 An Empirical Study on Web Mining of Parallel Data Gumwon Hong1, Chi-Ho Li2, Ming Zhou2 and Hae-Chang Rim1 1Department of Computer Science & En- gineering, Korea University {gwhong,rim}@nlp.korea.ac.kr 2Natural Language Computing Group, Microsoft Research Asia {chl,mingzhou}@microsoft.com Abstract This paper 1 presents an empirical ap- proach to mining parallel corpora. Con- ventional approaches use a readily available collection of comparable, non- parallel corpora to extract parallel sen- tences. This paper attempts the much more challenging task of directly search- ing for high-quality sentence pairs from the Web. We tackle the problem by formulating good search query using „Learning to Rank‟ and by filtering noisy document pairs using IBM Model 1 alignment. End-to-end evaluation shows that the proposed approach sig- nificantly improves the performance of statistical machine translation. 1 Introduction Bilingual corpora are very valuable resources in NLP. They can be used in statistical machine translation (SMT), cross language information retrieval, and paraphrasing. Thus the acquisition of bilingual corpora has received much attention. Hansards, or parliamentary proceedings in more than one language, are obvious source of bilingual corpora, yet they are about a particular domain and therefore of limited use. Many re- searchers then explore the Web. Some approach attempts to locate bilingual text within a web page (Jiang et al., 2009); some others attempt to collect web pages in different languages and decide the parallel relationship between the web pages by means of structural cues, like exist- ence of a common ancestor web page, similarity between URLs, and similarity between the HTML structures (Chen and Nie, 2000; Resnik 1 This work has been done while the first author was visit- ing Microsoft Research Asia. and Smith, 2003; Yang and Li, 2003; Shi et al., 2006). The corpora thus obtained are generally of high quality and wide variety in domain, but the amount is still limited, as web pages that exhibit those structural cues are not abundant. Some other effort is to mine bilingual corpora by textual means only. That is, two pieces of text are decided to be parallel merely from the linguistic perspective, without considering any hint from HTML markup or website structure. These approaches (Zhao and Vogel, 2002; Utiyama and Isahara 2003; Fung and Cheung, 2004; Munteanu and Marcu, 2005; Abdul-Rauf and Schwenk, 2009) share roughly the same framework: Phase 1: Document Pair Retrieval 1) documents in some target language (TL) are stored in some database; 2) each document in some source language (SL) is represented by some TL keywords; 3) the TL keywords in (2) are used to assign some TL documents to a particular SL doc- ument, using some information retrieval (IR) technique. For example, Munteanu and Mar- cu (2005) apply the Lemur IR toolkit, Utiyama and Isahara (2003) use the BM25 similarity measure, and Fung and Cheung (2004) use cosine similarity. Each TL docu- ment pairs up with the SL document to form a candidate parallel document pair. Phase 2: …

Content truncated to 500 words