Learning Crosslingual Word Embeddings without Bilingual Corpora

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1285–1295, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning Crosslingual Word Embeddings without Bilingual Corpora Long Duong,12 Hiroshi Kanayama,3 Tengfei Ma,3 Steven Bird14 and Trevor Cohn1 1Department of Computing and Information Systems, University of Melbourne 2National ICT Australia, Victoria Research Laboratory 3IBM Research – Tokyo 4International Computer Science Institute, University of California Berkeley Abstract Crosslingual word embeddings represent lexi- cal items from different languages in the same vector space, enabling transfer of NLP tools. However, previous attempts had expensive re- source requirements, difficulty incorporating monolingual data or were unable to handle polysemy. We address these drawbacks in our method which takes advantage of a high coverage dictionary in an EM style training algorithm over monolingual corpora in two languages. Our model achieves state-of-the- art performance on bilingual lexicon induction task exceeding models using large bilingual corpora, and competitive results on the mono- lingual word similarity and cross-lingual doc- ument classification task. 1 Introduction Monolingual word embeddings have had widespread success in many NLP tasks includ- ing sentiment analysis (Socher et al., 2013), dependency parsing (Dyer et al., 2015), machine translation (Bahdanau et al., 2014). Crosslingual word embeddings are a natural extension facilitating various crosslingual tasks, e.g. through transfer learning. A model built in a source resource-rich language can then applied to the target resource poor languages (Yarowsky and Ngai, 2001; Das and Petrov, 2011; T¨ackstr¨om et al., 2012; Duong et al., 2015). A key barrier for crosslingual transfer is lexical matching between the source and the target language. Crosslingual word embeddings are a natural remedy where both source and target language lexicon are presented as dense vectors in the same vector space (Klementiev et al., 2012). Most previous work has focused on down-stream crosslingual applications such as document classi- fication and dependency parsing. We argue that good crosslingual embeddings should preserve both monolingual and crosslingual quality which we will use as the main evaluation criterion through mono- lingual word similarity and bilingual lexicon induc- tion tasks. Moreover, many prior work (Chandar A P et al., 2014; Koˇcisk´y et al., 2014) used bilingual or comparable corpus which is also expensive for many low-resource languages. Søgaard et al. (2015) im- pose a less onerous data condition in the form of linked Wikipedia entries across several languages, however this approach tends to underperform other methods. To capture the monolingual distributional properties of words it is crucial to train on large monolingual corpora (Luong et al., 2015). How- ever, many previous approaches are not capable of scaling up either because of the complicated objec- tive functions or the nature of the algorithm. Other methods use a dictionary as the bridge between lan- guages (Mikolov et al., 2013a; Xiao and Guo, 2014), however they do not adequately handle translation ambiguity. Our model uses a bilingual dictionary from Pan- lex (Kamholz et al., 2014) as the source of bilin- gual signal. Panlex covers more than a thousand lan- guages and …

Content truncated to 500 words