A DOM Tree Alignment Model for Mining Parallel Data from the Web

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 489–496, Sydney, July 2006. c⃝2006 Association for Computational Linguistics A DOM Tree Alignment Model for Mining Parallel Data from the Web Lei Shi1, Cheng Niu1, Ming Zhou1, and Jianfeng Gao2 1Microsoft Research Asia, 5F Sigma Center, 49 Zhichun Road, Beijing 10080, P. R. China 2Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA {leishi,chengniu,mingzhou,jfgao}@microsoft.com Abstract This paper presents a new web mining scheme for parallel data acquisition. Based on the Document Object Model (DOM), a web page is represented as a DOM tree. Then a DOM tree alignment model is proposed to identify the transla- tionally equivalent texts and hyperlinks between two parallel DOM trees. By tracing the identified parallel hyperlinks, parallel web documents are recursively mined. Compared with previous mining schemes, the benchmarks show that this new mining scheme improves the mining coverage, reduces mining bandwidth, and enhances the quality of mined parallel sentences. 1 Introduction Parallel bilingual corpora are critical resources for statistical machine translation (Brown 1993), and cross-lingual information retrieval (Nie 1999). Additionally, parallel corpora have been exploited for various monolingual natural lan- guage processing (NLP) tasks, such as word- sense disambiguation (Ng 2003) and paraphrase acquisition (Callison 2005). However, large scale parallel corpora are not readily available for most language pairs. Even where resources are available, such as for Eng- lish-French, the data are usually restricted to government documents (e.g., the Hansard corpus, which consists of French-English translations of debates in the Canadian parliament) or newswire texts. The "governmentese" that characterizes these document collections cannot be used on its own to train data-driven machine translation sys- tems for a range of domains and language pairs. With a sharply increasing number of bilingual web sites, web mining for parallel data becomes a promising solution to this knowledge acquisi- tion problem. In an effort to estimate the amount of bilingual data on the web, (Ma and Liberman 1999) surveyed web pages in the de (German web site) domain, showing that of 150,000 web- sites in the .de domain, 10% are German-English bilingual. Based on such observations, some web mining systems have been developed to auto- matically obtain parallel corpora from the web (Nie et al 1999; Ma and Liberman 1999; Chen, Chau and Yeh 2004; Resnik and Smith 2003 � Zhang et al 2006 ). These systems mine parallel web documents within bilingual web sites, ex- ploiting the fact that URLs of many parallel web pages are named with apparent patterns to facili- tate website maintenance. Hence given a bilin- gual website, the mining systems use pre-defined URL patterns to discover candidate parallel documents within the site. Then content-based features will be used to verify the translational equivalence of the candidate pairs. However, due to the diversity of web page styles and website maintenance mechanisms, bilingual websites use varied naming schemes for parallel documents. For example, the United Nation’s website, which contains thousands of parallel pages, simply names the majority of …

Content truncated to 500 words