CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6490–6500 August 1–6, 2021. ©2021 Association for Computational Linguistics 6490 CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web Holger Schwenk and Guillaume Wenzek and Sergey Edunov Edouard Grave and Armand Joulin and Angela Fan Facebook AI {schwenk,guw,edunov,egrave,ajoulin,angelafan}@fb.com Abstract We show that margin-based bitext mining in a multilingual sentence space can be success- fully scaled to operate on monolingual cor- pora of billions of sentences. We use 32 Com- mon Crawl snapshots (Wenzek et al., 2019), to- talling 71 billion unique sentences. Using one unified approach for 90 languages, we were able to mine 10.8 billion parallel sentences, out of which only 2.9 billions are aligned with English. We illustrate the capability of our scalable mining system to create high qual- ity training sets from one language to any other by training hundreds of different ma- chine translation models and evaluating them on the many-to-many TED benchmark. Fur- ther, we evaluate on competitive translation benchmarks such as WMT and WAT. Using only mined bitext, we set a new state of the art for a single system on the WMT’19 test set for English-German/Russian/Chinese. In particu- lar, our English/German and English/Russian systems outperform the best single ones by over 4 BLEU points and are on par with best WMT’19 systems, which train on the WMT training data and augment it with backtrans- lation. We also achieve excellent results for distant languages pairs like Russian/Japanese, outperforming the best submission at the 2020 WAT workshop. All of the mined bitext will be freely available. 1 Introduction Parallel data, i.e. sentences in two languages which are mutual translations, are a crucial resource for many multilingual natural language processing tasks. Traditionally, high quality parallel texts are obtained from the publications of international or- ganizations like the the United Nations (Ziemski et al., 2016) or the European Parliament (Koehn, 2005). These are professional human translations, but they are in a more formal language and tend to be limited to political topics. Another direction is to rely on volunteers to provide translations for public texts, such as the TED corpus (Qi et al., 2018), news commentary (Tiedemann, 2012) or OpenSubtitles (Lison and Tiedemann, 2016), but this approach lacks scalability. There is also a large body of works which aims in mining bitexts by comparing huge collections of monolingual data. Our aim is to mine at mas- sive scale, both in number of possible languages and in quantity of mined parallel sentences. Most existing large scale bitext mining techniques use a hierarchical approach. First, a subset of texts that may contain parallel sentences are selected at the document level. Subsequently, sentences within these aligned documents are compared to identify parallel ones. This local mining is potentially fast since only a few thousand sentences need to be compared for each document pair. However, sen- tences not present in these pre-selected documents cannot be aligned, which …

Content truncated to 500 words