WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1351–1361 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1351 WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia Holger Schwenk Facebook AI schwenk@fb.com Vishrav Chaudhary Facebook AI vishrav@fb.com Shuo Sun Johns Hopkins University ssun32@jhu.edu Hongyu Gong Univ. of Illinois Urbana-Champaign hgong6 @illinois.edu Francisco Guzm´an Facebook AI fguzman@fb.com Abstract We present an approach based on multilin- gual sentence embeddings to automatically ex- tract parallel sentences from the content of Wikipedia articles in 96 languages, including several dialects or low-resource languages. We systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus is freely available.1 To get an indication on the quality of the ex- tracted bitexts, we train neural MT baseline systems on the mined data only for 1886 lan- guages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English. 1 Introduction Most of the current approaches in Natural Lan- guage Processing are data-driven. The size of the resources used for training is often the primary con- cern, but the quality and a large variety of topics may be equally important. Monolingual texts are usually available in huge amounts for many topics and languages. However, multilingual resources, i.e. sentences which are mutual translations, are more limited, in particular when the two languages do not involve English. An important source of parallel texts is from international organizations like the European Parliament (Koehn, 2005) or the United Nations (Ziemski et al., 2016). Several projects rely on volunteers to provide translations for public texts, e.g. news commentary (Tiede- mann, 2012), OpensubTitles (Lison and Tiede- mann, 2016) or the TED corpus (Qi et al., 2018) 1https://github.com/facebookresearch/ LASER/tree/master/tasks/WikiMatrix Wikipedia is probably the largest free multi- lingual resource on the Internet. The content of Wikipedia is very diverse and covers many topics. Articles exist in more than 300 languages. Some content on Wikipedia was human translated from an existing article into another language, not neces- sarily from or into English. Eventually, the trans- lated articles have been later independently edited and are not parallel anymore. Wikipedia strongly discourages the use of unedited machine transla- tion,2 but the existence of such articles cannot be totally excluded. Many articles have been written independently, but may nevertheless contain sen- tences which are mutual translations. This makes Wikipedia a very appropriate resource to mine for parallel texts for a large number of language pairs. To the best of our knowledge, this is the first work to process the entire Wikipedia and systematically mine for parallel sentences in all language pairs. In this work, we build on a recent approach to mine parallel texts based on a distance measure in …

Content truncated to 500 words