Crawling Back and Forth: Using Back and Out Links to Locate Bilingual Sites

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 429–437, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Crawling Back and Forth: Using Back and Out Links to Locate Bilingual Sites Luciano Barbosa AT&T Labs – Research 180 Park Ave Florham Park, NJ 07932 lbarbosa@research.att.com Srinivas Bangalore AT&T Labs – Research 180 Park Ave Florham Park, NJ 07932 srini@research.att.com Vivek Kumar Sridhar Rangarajan AT&T Labs – Research 180 Park Ave Florham Park, NJ 07932 vkumar@research.att.com Abstract This paper presents a novel crawling strat- egy to locate bilingual sites. It does so by focusing on the Web graph neighbor- hood of these sites and exploring the pat- terns of the links in this region to guide its visitation policy. A sub-task in the prob- lem of bilingual site discovery is the job of detecting bilingual sites, i.e., given a Web site, verify whether it is bilingual or not. We perform this task by combining supervised learning and language iden- tification. Experimental results demon- strate that our crawler outperforms previ- ous crawling approaches and produces a high-quality collection of bilingual sites, which we evaluate in the context of ma- chine translation in the tourism and hos- pitality domain. The parallel text obtained using our novel crawling strategy results in a relative improvement of 22% in BLEU score (English-to-Spanish) over an out-of- domain seed translation model trained on the European parliamentary proceedings. 1 Introduction Parallel texts are translations of the same text in different languages. Parallel text acquisition from the Web has received increased attention in the recent years, especially for machine transla- tion (Melamed, 2001) and cross-language infor- mation retrieval (Grossman and Frieder, 2004). For many years, the European Parliament proceed- ings (Koehn, 2005a) and official documents of countries with multiple languages were the only widely available parallel texts. Although these are high-quality corpora, they have some limitations: (1) they tend to be domain specific (e.g., govern- ment related texts); (2) they are available in only a few languages; and (3) sometimes they are not free or there is some restriction for using them. On the other hand, Web data is free and comprises data from different languages and domains. Previous research in the area of parallel Web data acquisition has mainly focused on the prob- lems of document pair identification (Jiang et al., 2009; Uszkoreit et al., 2010; Munteanu and Marcu, 2005; Resnik and Smith, 2003; Melamed, 2001) and sentence alignment. Typically, doc- ument pairs are located by issuing queries to a search engine (Resnik and Smith, 2003; Hong et al., 2010). The sentences in the matched doc- uments are then aligned using standard dynamic programming techniques. In this work, we model the problem of obtaining parallel text in two sub- tasks. First, locate the sites that contain bilingual data (bilingual sites). Here we assume that paral- lel texts are present in the same site (Chen and Nie, 2000). Second, extract parallel texts within these sites. While the latter problem of extracting of par- allel text …

Content truncated to 500 words