On the Role of Seed Lexicons in Learning Bilingual Word Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 247–257, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics On the Role of Seed Lexicons in Learning Bilingual Word Embeddings Ivan Vuli´c and Anna Korhonen Language Technology Lab DTAL, University of Cambridge {iv250, alk23}@cam.ac.uk Abstract A shared bilingual word embedding space (SBWES) is an indispensable resource in a variety of cross-language NLP and IR tasks. A common approach to the SB- WES induction is to learn a mapping func- tion between monolingual semantic spaces, where the mapping critically relies on a seed word lexicon used in the learning pro- cess. In this work, we analyze the impor- tance and properties of seed lexicons for the SBWES induction across different di- mensions (i.e., lexicon source, lexicon size, translation method, translation pair relia- bility). On the basis of our analysis, we propose a simple but effective hybrid bilin- gual word embedding (BWE) model. This model (HYBWE) learns the mapping be- tween two monolingual embedding spaces using only highly reliable symmetric trans- lation pairs from a seed document-level embedding space. We perform bilingual lexicon learning (BLL) with 3 language pairs and show that by carefully selecting reliable translation pairs our new HYBWE model outperforms benchmarking BWE learning models, all of which use more expensive bilingual signals. Effectively, we demonstrate that a SBWES may be in- duced by leveraging only a very weak bilin- gual signal (document alignments) along with monolingual data. 1 Introduction Dense real-valued vector representations of words or word embeddings (WEs) have recently gained increasing popularity in natural language process- ing (NLP), serving as invaluable features in a broad Monolingual vs Bilingual Figure 1: A toy example of a 3-dimensional mono- lingual vs shared bilingual word embedding space (further SBWES) from Gouws et al. (2015). range of NLP tasks, e.g., (Turian et al., 2010; Col- lobert et al., 2011; Chen and Manning, 2014). Sev- eral studies have showcased a direct link and com- parable performance to “more traditional” distribu- tional models (Turney and Pantel, 2010). Yet the widely used skip-gram model with negative sam- pling (SGNS) (Mikolov et al., 2013b) is considered as the state-of-the-art word representation model, due to its simplicity, fast training, as well as its solid and robust performance across a wide variety of semantic tasks (Baroni et al., 2014; Levy and Goldberg, 2014b; Levy et al., 2015). Research interest has recently extended to bilin- gual word embeddings (BWEs). BWE learning models focus on the induction of a shared bilingual word embedding space (SBWES) where words from both languages are represented in a uniform language-independent manner such that similar words (regardless of the actual language) have sim- ilar representations (see Fig. 1). A variety of BWE learning models have been proposed, differing in the essential requirement of a bilingual signal nec- essary to construct such a SBWES (discussed later in Sect. 2). SBWES may be used to support many tasks, e.g., computing cross-lingual/multilingual semantic word similarity (Faruqui and Dyer, 2014), learning bilingual word …

Content truncated to 500 words