Translation Invariant Word Embeddings

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1084–1088, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Translation Invariant Word Embeddings Matt Gardner∗ Carnegie Mellon University mg1@cs.cmu.edu Kejun Huang∗ University of Minnesota huang663@umn.edu Evangelos Papalexakis Carnegie Mellon University epapalex@cs.cmu.edu Xiao Fu University of Minnesota xfu@umn.edu Partha Talukdar Indian Institute of Science ppt@serc.iisc.in Christos Faloutsos Carnegie Mellon University christos@cs.cmu.edu Nicholas Sidiropoulos University of Minnesota nikos@umn.edu Tom Mitchell Carnegie Mellon University tom.mitchell@cmu.edu Abstract This work focuses on the task of finding latent vector representations of the words in a corpus. In particular, we address the issue of what to do when there are multiple languages in the corpus. Prior work has, among other techniques, used canonical correlation analysis to project pre-trained vectors in two languages into a common space. We propose a simple and scal- able method that is inspired by the no- tion that the learned vector representations should be invariant to translation between languages. We show empirically that our method outperforms prior work on mul- tilingual tasks, matches the performance of prior work on monolingual tasks, and scales linearly with the size of the input data (and thus the number of languages be- ing embedded). 1 Introduction Representing words as vectors in some latent space has long been a central idea in natural lan- guage processing. The distributional hypothesis, perhaps best stated as “You shall know a word by the company it keeps” (Firth, 1957), has had a long and productive history, as well as a recent revival in neural-network-based models (Mikolov et al., 2013). These methods generally construct a word by context matrix, then either use the vectors di- rectly (often weighted by term frequency and in- verse document frequency), perform some factor- ∗These authors contributed equally. ization of the matrix, or use it as input to a neu- ral network which produces vectors for each word. The resultant vectors can be used in a wide array of tasks, from information retrieval to part-of-speech tagging and parsing. There has also been some recent work address- ing how to create these vectors when informa- tion from multiple languages is available. Two recent attempts involve using canonical correla- tion analysis (CCA) to project pre-trained vec- tors from each of two languages into a common space (Faruqui and Dyer, 2014b) and using an alignment matrix to heuristically project the vec- tors from one language onto the words in another language (Guo et al., 2015). These methods gener- ally only work with two languages at a time, how- ever. In this paper, we introduce a technique for con- structing multilingual word embeddings that is in- spired by the notion of translational invariance. CCA and the heuristic projection mentioned above both attempt to construct vectors such that words that are translations of each other are close in the vector space, but the method we introduce formal- izes this as part of the objective function of the original decomposition. We further show how to optimize this objective function …

Content truncated to 500 words