Trans-gram, Fast Cross-lingual Word-embeddings

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1109–1113, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Trans-gram, Fast Cross-lingual Word-embeddings reyes − Mannde = reginait − femmefr Jocelyn Coulmance 105 rue La Fayette 75010 Paris joc@proxem.com Jean-Marc Marty ∗ 105 rue La Fayette 75010 Paris jmm@proxem.com Guillaume Wenzek ∗ 105 rue La Fayette 75010 Paris guw@proxem.com Amine Benhalloum 105 rue La Fayette 75010 Paris aba@proxem.com Abstract We introduce Trans-gram, a simple and computationally-efficient method to simultaneously learn and align word- embeddings for a variety of languages, us- ing only monolingual data and a smaller set of sentence-aligned data. We use our new method to compute aligned word- embeddings for twenty-one languages us- ing English as a pivot language. We show that some linguistic features are aligned across languages for which we do not have aligned data, even though those properties do not exist in the pivot language. We also achieve state of the art results on standard cross-lingual text classification and word translation tasks. 1 Introduction Word-embeddings are a representation of words with fixed-sized vectors. It is a distributed rep- resentation (Hinton, 1984) in the sense that there is not necessarily a one-to-one correspondence be- tween vector dimensions and linguistic properties. The linguistic properties are distributed along the dimensions of the space. A popular method to compute word- embeddings is the Skip-gram model (Mikolov et al., 2013a). This algorithm learns high-quality word vectors with a computation cost much lower than previous methods. This allows the processing of very important amounts of data. For instance, a 1.6 billion words dataset can be processed in less than one day. Several authors came up with different methods to align word-embeddings across two languages (Klementiev et al., 2012; Mikolov et al., 2013b; Lauly et al., 2014; Gouws et al., 2015). ∗These authors contributed equally. In this article, we introduce a new method called Trans-gram, which learns word embed- dings aligned across many languages, in a simple and efficient fashion, using only sentence align- ments rather than word alignments. We compare our method with previous approaches on a cross- lingual document classification task and on a word translation task and obtain state of the art results on these tasks. Additionally, word-embeddings for twenty-one languages are learned simultaneously - to our knowledge - for the first time, in less than two and a half hours. Furthermore, we illustrate some interesting properties that are captured such as cross-lingual analogies, e.g ⃗reyes − ⃗ Mannde + ⃗ femmefr ≈ ⃗ reginait which can be used for dis- ambiguation. 2 Review of Previous Work A number of methods have been explored to train and align bilingual word-embeddings. These methods pursue two objectives: first, similar rep- resentations (i.e. spatially close) must be assigned to similar words (i.e. “semantically close”) within each language - this is the mono-lingual objec- tive; second, similar representations must be as- signed to similar words across languages - this is the cross-lingual objective. The simplest approach consists in …

Content truncated to 500 words