Harvesting Parallel Text in Multiple Languages with Limited Supervision

Proceedings of COLING 2012: Technical Papers, pages 201–214, COLING 2012, Mumbai, December 2012. Harvesting parallel text in multiple languages with limited supervision Luciano Barbosa1 V ivek Kumar Rangarajan Sridhar1 Mahsa Y armohammadi2 Srinivas Bangalore1 (1) AT&T Labs - Research, 180 Park Avenue, Florham Park, New Jersey 07932 (2) Oregon Health & Science University, Beaverton, Oregon 97006 lbarbosa,vkumar,srini@research.att.com, yarmoham@ohsu.edu Abstract The Web is an ever increasing, dynamically changing, multilingual repository of text. There have been several approaches to harvest this repository for bootstrapping, supplementing and adapting data needed for training models in speech and language applications. In this paper, we present semi-supervised and unsupervised approaches to harvesting multilingual text that rely on a key observation of link collocation. We demonstrate the effectiveness of our approach in the context of statistical machine translation by harvesting parallel texts and training translation models in 20 different languages. Furthermore, by exploiting the DOM trees of parallel webpages, we extend our harvesting technique to create parallel data for resource limited languages in an unsupervised manner. We also present some interesting observations concerning the socio-economic factors that the multilingual Web reflects. Keywords: web crawling, parallel text, document model object tree, machine translation. 1 Introduction The amount of information and knowledge in the World Wide Web (WWW) is increasing at a rapid rate. As of September 2011, approximately 500 million websites were estimated to be present in the WWW; a jump of 1000% from 1995 1. An increasingly large proportion of these websites are in non-English languages. For example, the total proportion of English webpages on the WWW has been estimated to have dropped from 80% in 1996 to about 45% in 2008 (Pimienta et al., 2009). Furthermore, only 27% of the internet users claim English as their native language (Miniwatts Marketing Group, 2011). The opportunity to capitalize on this new market has encouraged internet service providers to provide web content in multiple languages. As a result, the Web has become an attractive resource for acquiring structured and unstructured data across several low-resource languages and domains. Multilingual web data has been especially useful in a variety of natural language processing tasks such as information retrieval, language modeling and machine translation. Statistical machine translation requires the creation of a large corpus of parallel text, i.e., translations of text across languages. Harvesting such data automatically by exploiting multilingual webpages provides a low-cost, scalable alternative to expensive expert human translations. Moreover, bilingual subject matter experts may be extremely difficult to find for certain 1http://news.netcraft.com/archives/2011/09/06/september-2011-web-server-survey.html 201 language pairs. As a result, web harvesting of parallel text has been addressed extensively in the recent past (Resnik and Smith, 2003; Shi et al., 2006; Pomikálek, 2008; Utiyama et al., 2009; Hong et al., 2010; Almeida and Simões, 2010; Uszkoreit et al., 2010). Parallel text acquisition can be performed by examining the Web in either an unstructured or structured way. In the unstructured view of the Web, a large Web index is used as a starting point and the webpages are matched using cross-language document …

Content truncated to 500 words