Mining Parallel Documents Using Low Bandwidth and High Precision CLIR from the Heterogeneous Web

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 420–428, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Mining Parallel Documents Using Low Bandwidth and High Precision CLIR from the Heterogeneous Web Simon Shi1, Pascale Fung1, Emmanuel Prochasson2, Chi-kiu Lo1 and Dekai Wu1 1 Human Language Technology Center Hong Kong University of Science and Technology (HKUST) Clear Water Bay, Hong Kong 2 Digital Butter 69 Jervois Street, Sheung Wan, Hong Kong {eesys,pascale}@ust.hk, emmanuel@butter.com.hk, {jackielo,dekai}@cs.ust.hk Abstract We propose a content-based approach to mine parallel resources from the entire web using cross lingual information re- trieval (CLIR) with search query rele- vance score (SQRS). Our method im- proves mining recall by going beyond URL matching to find parallel documents from non-parallel sites. We introduce SQRS to improve the precision of min- ing. Our method makes use of search en- gines to query for target document given each source document and therefore does not require downloading target language documents in batch mode, reducing com- putational cost on the local machines and bandwidth consumption. We obtained a very high mining precision (88%) on the parallel documents by the pure CLIR ap- proach. After extracting parallel sentenc- es from the mined documents and using them to train an SMT system, we found that the SMT performance, with 29.88 BLEU score, is comparable to that ob- tained with high quality manually trans- lated parallel sentences with 29.54 BLEU score, illustrating the excellent quality of the mined parallel material. 1 Introduction Parallel resources such as bilingual lexicon and sentence translations are typically obtained from translated parallel documents. The web has now grown into an archive of trillions of URLs, het- erogeneous in nature, in the last decade. There is a need to readdress the problem of how to mine parallel documents from the web. We suggest that parallel documents can be mined with high precision from web sites that are not necessarily parallel to each other. Parallel resources reside on a diverse range of websites which can be classified into the follow- ing categories: Parallel websites: single website with struc- turally aligned bilingual pages. Typically they are websites of institutions, governments and commercial companies. (e.g. Financial Times Chinese/English, Wall Street Journal Chi- nese/English). Structure based methods were previously proposed to mine parallel documents from these websites: Resnik and Smith (2003) used (1) parent pages containing links to versions of one document in different languages and (2) sibling pages con- tains link to translation of the current documents. They also rely on the URL and anchor text to spot language specific version of documents. A structural alignment using DOM tree repre- sentation was proposed by Shi et al. (2006) to align parallel documents by using HTML struc- ture. They identify the translational equivalent texts and hyperlinks between two parallel DOM trees to find parallel documents. However, the web is a heterogeneous collec- tion of documents that extend far beyond bilin- gual and comparable pages with obvious struc- tural features, such as similar URLs …

Content truncated to 500 words