word2word: A Collection of Bilingual Lexicons for 3,564 Language Pairs

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3036–3045 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3036 word2word: A Collection of Bilingual Lexicons for 3,564 Language Pairs Yo Joong Choe*, Kyubyong Park*, Dongwoo Kim* Kakao Brain 20, Pangyoyeok-ro 241, Bundang-gu, Seongnam-si, Gyeonggi-do, Korea {yj.choe,kyubyong.park,dongwoo.kim}@kakaobrain.com Abstract We present word2word, a publicly available dataset and an open-source Python package for cross-lingual word translations extracted from sentence-level parallel corpora. Our dataset provides top-k word translations in 3,564 (directed) language pairs across 62 languages in OpenSubtitles2018 (Lison et al., 2018). To obtain this dataset, we use a count-based bilingual lexicon extraction model based on the observation that not only source and target words but also source words themselves can be highly correlated. We illustrate that the resulting bilingual lexicons have high coverage and attain competitive translation quality for several language pairs. We wrap our dataset and model in an easy-to-use Python library, which supports downloading and retrieving top-k word translations in any of the supported language pairs as well as computing top-k word translations for custom parallel corpora. Keywords: bilingual lexicon, word translation, Python toolkit 1. Introduction Bilingual lexicons (Fung, 1998) are valuable resources for cross-lingual tasks, including low-resource machine trans- lation (Ramesh and Sankaranarayanan, 2018; G¯u et al., 2019) and cross-lingual word embeddings (Ruder et al., 2017). However, it is often difficult to find a large enough set of bilingual lexicons that is freely and readily available across various language pairs (Levy et al., 2017). For exam- ple, standard bilingual dictionaries like Wiktionary1 often do not explicitly provide word correspondences but refers or redirects to the query word’s dictionary form: • Query: travaill´e (French for ‘worked’) Result: (verb) past principle of travailler ‘work’ • Query: ᄆ ᅥ ᆨᄋ ᅥ ᆻᄃ ᅡ (Korean for ‘ate’) Result: redirects to ᄆ ᅥ ᆨᄃ ᅡ ‘eat’ Not only does this make it tedious to find word-level cor- respondences across many query words, this is particularly problematic when we try to find word correspondences for languages where some dictionary forms are rarely used in ordinary discourse, such as the case of ᄆ ᅥ ᆨᄃ ᅡ in the Korean language. While the task of bilingual lexicon extraction (BLE) has been popular in both early and recent literature, span- ning from count-based approaches (Fung, 1998; Vuli´c and Moens, 2013; Liu et al., 2013) to using cross-lingual word embeddings (Ruder et al., 2017; Mikolov et al., 2013a; Gouws et al., 2015; Conneau et al., 2017; Levy et al., 2017; Artetxe et al., 2018; Artetxe et al., 2019), few were focused on building high-coverage bilingual lexicons across many language pairs, possibly including non-Indo-European lan- guages. In fact, many of the recent studies and their ac- companying packages (Conneau et al., 2017; Artetxe et al., 2018; Glavaˇs et al., 2019) aim at evaluating cross-lingual word embeddings, so that they involve at most 10-100s of language pairs and 1-5K words for each pair. *Equal contribution. 1https://en.wiktionary.org # Languages 62 # Language Pairs 3,564 …

Content truncated to 500 words