Mining Bilingual Data from the Web with Adaptively Learnt Patterns

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 870–878, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Mining Bilingual Data from the Web with Adaptively Learnt Patterns Long Jiang1, Shiquan Yang2, Ming Zhou1, Xiaohua Liu1, Qingsheng Zhu2 1Microsoft Research Asia Beijing, 100190, P.R.China 2Chongqing University, Chongqing, 400044, P.R.China {longj,mingzhou,xiaoliu}@microsoft.com shiquany@gmail.com,qszhu@cqu.edu.cn Abstract Mining bilingual data (including bilingual sen- tences and terms1) from the Web can benefit many NLP applications, such as machine translation and cross language information re- trieval. In this paper, based on the observation that bilingual data in many web pages appear collectively following similar patterns, an adaptive pattern-based bilingual data mining method is proposed. Specifically, given a web page, the method contains four steps: 1) pre- processing: parse the web page into a DOM tree and segment the inner text of each node into snippets; 2) seed mining: identify poten- tial translation pairs (seeds) using a word based alignment model which takes both trans- lation and transliteration into consideration; 3) pattern learning: learn generalized patterns with the identified seeds; 4) pattern based min- ing: extract all bilingual data in the page using the learned patterns. Our experiments on Chi- nese web pages produced more than 7.5 mil- lion pairs of bilingual sentences and more than 5 million pairs of bilingual terms, both with over 80% accuracy. 1 Introduction Bilingual data (including bilingual sentences and bilingual terms) are critical resources for build- ing many applications, such as machine transla- tion (Brown, 1993) and cross language informa- tion retrieval (Nie et al., 1999). However, most existing bilingual data sets are (i) not adequate for their intended uses, (ii) not up-to-date, (iii) apply only to limited domains. Because it‟s very hard and expensive to create a large scale bilin- 1 In this paper terms refer to proper nouns, technical terms, movie names, and so on. And bilingual terms/sentences mean terms/sentences and their translations. gual dataset with human effort, recently many researchers have turned to automatically mining them from the Web. If the content of a web page is written in two languages, we call the page a Bilingual Web Page. Many such pages exist in non-English web sites. Most of them have a primary language (usually a non-English language) and a second- ary language (usually English). The content in the secondary language is often the translation of some primary language text in the page. Since bilingual web pages are very common in non-English web sites, mining bilingual data from them should be an important task. However, as far as we know, there is no publication availa- ble on mining bilingual sentences directly from bilingual web pages. Most existing methods for mining bilingual sentences from the Web, such as (Nie et al., 1999; Resnik and Smith, 2003; Shi et al., 2006), try to mine parallel web documents within bilingual web sites first and then extract bilingual sentences from mined parallel docu- ments using sentence alignment methods. As to mining term translations from …

Content truncated to 500 words