CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5960–5969, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5960 CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs Ahmed El-Kishky1 Vishrav Chaudhary1 Francisco Guzmán1 Philipp Koehn2 1Facebook AI 2Johns Hopkins University {ahelk,vishrav,fguzman}@fb.com, phi@jhu.edu Abstract Cross-lingual document alignment aims to identify pairs of documents in two distinct lan- guages that are of comparable content or trans- lations of each other. In this paper, we ex- ploit the signals embedded in URLs to label web documents at scale with an average preci- sion of 94.5% across different language pairs. We mine sixty-eight snapshots of the Com- mon Crawl corpus and identify web document pairs that are translations of each other. We release a new web dataset consisting of over 392 million URL pairs from Common Crawl covering documents in 8144 language pairs of which 137 pairs include English. In addi- tion to curating this massive dataset, we in- troduce baseline methods that leverage cross- lingual representations to identify aligned doc- uments based on their textual content. Finally, we demonstrate the value of this parallel doc- uments dataset through a downstream task of mining parallel sentences and measuring the quality of machine translations from models trained on this mined data. Our objective in releasing this dataset is to foster new research in cross-lingual NLP across a variety of low, medium, and high-resource languages. 1 Introduction Cross-lingual document alignment aims to pair documents such that they are translations or near translations of each other. There are a variety of tasks in natural language processing that con- sume such parallel cross-lingual data. Tradition- ally, machine translation approaches have lever- aged parallel sentences as training data for use with sequence-to-sequence models. Other tasks include cross-lingual information retrieval and cross-lingual document classification. Addition- ally, cross-lingual data facilitates training cross- lingual representations such as multilingual BERT (Devlin et al., 2019) and XLM (Lample and Con- neau, 2019) which are used in many NLP tasks. The availability of high-quality datasets is neces- sary to both train and evaluate models across these many tasks. While it is possible to manually label aligned documents across languages, the process is costly and time consuming due to the quadratic search space for document pairs. Additionally, for low-resource languages, identifying these cross- lingual document pairs is more difficult due to their relative scarcity. Furthermore, lack of access to qualified human annotators makes it necessary to have additional quality control in low-resource scenarios (Guzmán et al., 2019). In this paper, we investigate whether we can rely on weak supervision to generate labels for document pairs. In particular, we focus on the weak signals embedded in the URLs of web doc- uments, that can be used to identify the differ- ent translations of a single document across many languages. We preprocess, filter, and apply a set of high-precision hand-crafted rules to automati- cally sift through a massive collection of 169 bil- lion web documents and identify over a 392 mil- lion cross-lingual parallel …

Content truncated to 500 words