Learning bilingual word embeddings with (almost) no bilingual data

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 451–462 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1042 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 451–462 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1042 Learning bilingual word embeddings with (almost) no bilingual data Mikel Artetxe Gorka Labaka Eneko Agirre IXA NLP group University of the Basque Country (UPV/EHU) {mikel.artetxe,gorka.labaka,e.agirre}@ehu.eus Abstract Most methods to learn bilingual word em- beddings rely on large parallel corpora, which is difficult to obtain for most lan- guage pairs. This has motivated an ac- tive research line to relax this requirement, with methods that use document-aligned corpora or bilingual dictionaries of a few thousand words instead. In this work, we further reduce the need of bilingual re- sources using a very simple self-learning approach that can be combined with any dictionary-based mapping technique. Our method exploits the structural similarity of embedding spaces, and works with as little bilingual evidence as a 25 word dictionary or even an automatically generated list of numerals, obtaining results comparable to those of systems that use richer resources. 1 Introduction Multilingual word embeddings have attracted a lot of attention in recent times. In addition to having a direct application in inherently crosslingual tasks like machine translation (Zou et al., 2013) and crosslingual entity linking (Tsai and Roth, 2016), they provide an excellent mechanism for transfer learning, where a model trained in a resource-rich language is transferred to a less-resourced one, as shown with part-of-speech tagging (Zhang et al., 2016), parsing (Xiao and Guo, 2014) and docu- ment classification (Klementiev et al., 2012). Most methods to learn these multilingual word embeddings make use of large parallel corpora (Gouws et al., 2015; Luong et al., 2015), but there have been several proposals to relax this require- ment, given its scarcity in most language pairs. A possible relaxation is to use document-aligned or label-aligned comparable corpora (Søgaard et al., 2015; Vuli´c and Moens, 2016; Mogadala and Ret- tinger, 2016), but large amounts of such corpora are not always available for some language pairs. An alternative approach that we follow here is to independently train the embeddings for each language on monolingual corpora, and then learn a linear transformation to map the embeddings from one space into the other by minimizing the distances in a bilingual dictionary, usually in the range of a few thousand entries (Mikolov et al., 2013a; Artetxe et al., 2016). However, dictio- naries of that size are not readily available for many language pairs, specially those involving less-resourced languages. In this work, we reduce the need of large bilin- gual dictionaries to much smaller seed dictionar- ies. Our method can work with as little as 25 word pairs, which are straightforward to obtain assum- ing some basic knowledge of the languages in- volved. The method can also work with trivially generated seed dictionaries of numerals (i.e. 1-1, 2-2, 3-3, …

Content truncated to 500 words