ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555–4567 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4555 ParaCrawl: Web-Scale Acquisition of Parallel Corpora Marta Ba˜n´on†, Pinzhen Chen‡, Barry Haddow‡, Kenneth Heafield‡, Hieu Hoang‡ Miquel Espl`a-Gomis⋆, Mikel Forcada⋆, Amir Kamran♦, Faheem Kirefu‡ Philipp Koehn§, Sergio Ortiz-Rojas†, Leopoldo Pla⋆, Gema Ram´ırez-S´anchez† Elsa Sarr´ıas⋆, Marek Strelec‡, Brian Thompson§, William Waites‡, Dion Wiggins▲ Jaume Zaragoza† †Prompsit, ‡University of Edinburgh, ⋆University of Alicante §Johns Hopkins University, ♦TAUS, ▲Omniscien Technologies Abstract We report on methods to create the largest pub- licly available parallel corpora by crawling the web, using open source software. We empiri- cally compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems. 1 Introduction Parallel corpora are essential for building high- quality machine translation systems and have found uses in many other natural language ap- plications, such as learning paraphrases (Ban- nard and Callison-Burch, 2005; Hu et al., 2019) or cross-lingual projection of language tools (Yarowsky et al., 2001). We report on work to create the largest pub- licly available parallel corpora by crawling hun- dreds of thousands of web sites, using open source tools. The processing pipeline consists of the steps: crawling, text extraction, document align- ment, sentence alignment, and sentence pair fil- tering. We describe these steps in detail in Sec- tions 4–8. For some of these steps we evaluate sev- eral methods empirically in terms of their impact on machine translation quality. We provide the data resources used in these evaluations as bench- marks for future research. As part of these effort, several open source com- ponents have been developed. These are integrated into the open-source tool Bitextor,1 a highly mod- ular pipeline that allows harvesting parallel cor- pora from multilingual websites or from preexist- ing or historical web crawls such as the one avail- able as part of the Internet Archive.2 1https://github.com/bitextor/bitextor 2https://archive.org/ The execution of the pipeline has focused on of- ficial European Union languages, but also targeted Russian, Sinhala, Nepali, Tagalog, Swahili, and Somali. We show that the obtained parallel cor- pora improve state-of-the-art results on common benchmarks, such as the WMT Shared Task on News Translation. 2 Related Work While the idea of mining the web for parallel data has been already pursued in the 20th cen- tury (Resnik, 1999), the most serious efforts have been limited to large companies such as Google (Uszkoreit et al., 2010) and Microsoft (Rarrick et al., 2011), or targeted efforts on specific do- mains such as the Canadian Hansards and Eu- roparl (Koehn, 2005). The book Bitext Alignment (Tiedemann, 2011) describes some of the chal- lenges in greater detail. 2.1 Acquisition Efforts Most publicly available parallel corpora are the re- sult of targeted efforts to extract the translations from a specific source. The French–English Cana- dian Hansards3 were used in the earliest work on statistical machine translation. A …

Content truncated to 500 words