Simple task-specific bilingual word embeddings

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1386–1390, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Simple task-specific bilingual word embeddings∗ Stephan Gouws Stellenbosch University stephan@ml.sun.ac.za Anders Søgaard University of Copenhagen soegaard@hum.ku.dk Abstract We introduce a simple wrapper method that uses off-the-shelf word embedding algorithms to learn task-specific bilingual word em- beddings. We use a small dictionary of easily-obtainable task-specific word equiva- lence classes to produce mixed context-target pairs that we use to train off-the-shelf em- bedding models. Our model has the advan- tage that it (a) is independent of the choice of embedding algorithm, (b) does not re- quire parallel data, and (c) can be adapted to specific tasks by re-defining the equivalence classes. We show how our method outper- forms off-the-shelf bilingual embeddings on the task of unsupervised cross-language part- of-speech (POS) tagging, as well as on the task of semi-supervised cross-language super sense (SuS) tagging. 1 Introduction Using multi-layered neural networks to learn word embeddings has become standard in NLP (Turian et al., 2010; Guo et al., 2014). While there is still some controversy whether such methods are superior to older methods (Levy and Goldberg, 2014; Baroni et al., 2014), there is little doubt that continuous word representations can potentially solve some of the data sparsity problems inherent in NLP. Most research on word embeddings has focused on learning representations for the words in a sin- gle language, making syntactically or semantically similar words appear close in the embedding space. Embeddings have been applied to many tasks, from ∗The authors contributed equally to this work. The second author is funded by the ERC Starting Grant LOWLANDS No. 313695. named entity recognition (Turian et al., 2010) to de- pendency parsing (Bansal et al., 2014). It has fur- thermore been shown that weakly supervised em- bedding algorithms can also lead to huge improve- ments for tasks like sentiment analysis (Tang et al., 2014). In this work, we also use weak or distant su- pervision, relying on small dictionary seeds. This paper, however, considers the problem of learning bilingual word embeddings, i.e., word em- beddings such that similar words in two different languages end up close in the embedding space. Such bilingual word embeddings can potentially be used for better cross-language transfer of NLP mod- els, as we show in this paper. Previous work on bilin- gual word embeddings have defined similar words as translation equivalents and evaluated embeddings in the context of document classification tasks (Kle- mentiev et al., 2012; Kocisky et al., 2014). In this paper, we present a simple wrapper method to ex- isting monolingual word embedding algorithms that can learn task-specific bilingual embeddings, e.g., for POS tagging, named entity recognition, or sen- timent analysis. Our algorithm is simpler and per- forms better on the tasks where we could compare performance to existing algorithms. Also, we note that our approach, unlike existing algorithms (Kle- mentiev et al., 2012), is as fast as learning …

Content truncated to 500 words