Dirt Cheap Web-Scale Parallel Text from the Common Crawl

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1374–1383, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Dirt Cheap Web-Scale Parallel Text from the Common Crawl Jason R. Smith1,2 jsmith@cs.jhu.edu Philipp Koehn3 pkoehn@inf.ed.ac.uk Herve Saint-Amand3 herve@saintamh.org Chris Callison-Burch1,2,5 ccb@cs.jhu.edu ∗ Magdalena Plamada4 plamada@cl.uzh.ch Adam Lopez1,2 alopez@cs.jhu.edu 1Department of Computer Science, Johns Hopkins University 2Human Language Technology Center of Excellence, Johns Hopkins University 3School of Informatics, University of Edinburgh 4Institute of Computational Linguistics, University of Zurich 5Computer and Information Science Department, University of Pennsylvania Abstract Parallel text is the fuel that drives modern machine translation systems. The Web is a comprehensive source of preexisting par- allel text, but crawling the entire web is impossible for all but the largest compa- nies. We bring web-scale parallel text to the masses by mining the Common Crawl, a public Web crawl hosted on Amazon’s Elastic Cloud. Starting from nothing more than a set of common two-letter language codes, our open-source extension of the STRAND algorithm mined 32 terabytes of the crawl in just under a day, at a cost of about $500. Our large-scale experiment uncovers large amounts of parallel text in dozens of language pairs across a variety of domains and genres, some previously unavailable in curated datasets. Even with minimal cleaning and filtering, the result- ing data boosts translation performance across the board for five different language pairs in the news domain, and on open do- main test sets we see improvements of up to 5 BLEU. We make our code and data available for other researchers seeking to mine this rich new data resource.1 1 Introduction A key bottleneck in porting statistical machine translation (SMT) technology to new languages and domains is the lack of readily available paral- lel corpora beyond curated datasets. For a handful of language pairs, large amounts of parallel data ∗This research was conducted while Chris Callison- Burch was at Johns Hopkins University. 1github.com/jrs026/CommonCrawlMiner are readily available, ordering in the hundreds of millions of words for Chinese-English and Arabic- English, and in tens of millions of words for many European languages (Koehn, 2005). In each case, much of this data consists of government and news text. However, for most language pairs and do- mains there is little to no curated parallel data available. Hence discovery of parallel data is an important first step for translation between most of the world’s languages. text.The Web is an important source of parallel Many websites are available in multiple languages, and unlike other potential sources— such as multilingual news feeds (Munteanu and Marcu, 2005) or Wikipedia (Smith et al., 2010)— it is common to find document pairs that are di- rect translations of one another. This natural par- allelism simplifies the mining task, since few re- sources or existing corpora are needed at the outset to bootstrap the extraction process. Parallel text mining from the Web was origi- nally explored by individuals or small groups of academic researchers using search engines (Nie et al., …

Content truncated to 500 words