A Distribution-based Model to Learn Bilingual Word Embeddings

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1818–1827, Osaka, Japan, December 11-17 2016. A Distribution-based Model to Learn Bilingual Word Embeddings Hailong Cao1, Tiejun Zhao1, Shu Zhang2, Yao Meng2 1Harbin Institute of Technology, Harbin, China 2Fujitsu Research and Development Center, Beijing, China {hailong,tjzhao}@mtlab.hit.edu.cn {zhangshu,mengyao}@cn.fujitsu.com Abstract We introduce a distribution based model to learn bilingual word embeddings from monolingual data. It is simple, effective and does not require any parallel data or any seed lexicon. We take advantage of the fact that word embeddings are usually in form of dense real-valued low- dimensional vector and therefore the distribution of them can be accurately estimated. A novel cross-lingual learning objective is proposed which directly matches the distributions of word embeddings in one language with that in the other language. During the joint learning process, we dynamically estimate the distributions of word embeddings in two languages respectively and minimize the dissimilarity between them through standard back propagation algorithm. Our learned bilingual word embeddings allow to group each word and its translations together in the shared vector space. We demonstrate the utility of the learned embeddings on the task of finding word-to-word translations from monolingual corpora. Our model achieved encouraging performance on data in both related languages and substantially different languages. 1 Introduction Learning word vector representations based on neural network is now a ubiquitous technique in natural language processing tasks and applications. Tremendous advances have been brought by distributed rep- resentations to the state-of-the-art methods (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013a; Pennington et al., 2014). In these models, words are represented by dense real-valued low-dimensional vectors referred to as word embeddings learned from raw text. Distributed representa- tions have the property that similar words are represented by similar vectors and thus can achieve better generalization. Such representations are usually learned from monolingual data and therefore might not be generalized well across different languages. In order to learn useful syntactic and semantic features that are invariant to languages, several models for learning cross-lingual representations have been proposed and achieved impressive effects by incor- porating cross-lingual distributional information (Klementiev et al., 2012; Zou et al., 2013; Chandar et al., 2014; Faruqui and Dyer, 2014; Hermann and Blunsom, 2014; Gouws et al., 2015; Luong et al., 2015; Shi et al., 2015; Vuli´c and Moens, 2015; Upadhyay et al., 2016). In the cross-lingual settings, similar representations are desired for words denoting similar concepts in different languages (e.g., the embed- dings of the English word computer and the French word ordinateur should be similar). Cross-lingual representations are especially useful for many natural language processing tasks such as machine trans- lation (Zou et al., 2013; Zhang et al., 2014), computing cross-lingual word similarity (Zhang et al., 2016) and transferring knowledge from high-resource languages to low-resource languages (Guo et al., 2015), etc. However, all these cross-lingual models require some form of cross-lingual supervision such as seed lexicon,word-level alignments, sentence-level alignments and document-level alignments. Reliance on supervision might …

Content truncated to 500 words