Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1006–1011, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation Chao Xing CSLT, Tsinghua University Beijing Jiaotong University Beijing, P.R. China Chao Liu CSLT, RIIT, Tsinghua University CS Department, Tsinghua University Beijing, P.R. China Dong Wang* CSLT, RIIT, Tsinghua University TNList, China Beijing, P.R. China Yiye Lin CSLT, RIIT, Tsinghua University Beijing Institute of Technology Beijing, P.R. China Abstract Word embedding has been found to be high- ly powerful to translate words from one lan- guage to another by a simple linear transfor- m. However, we found some inconsistence among the objective functions of the embed- ding and the transform learning, as well as the distance measurement. This paper propos- es a solution which normalizes the word vec- tors on a hypersphere and constrains the lin- ear transform as an orthogonal transform. The experimental results confirmed that the pro- posed solution can offer better performance on a word similarity task and an English-to- Spanish word translation task. 1 Introduction Word embedding has been extensively studied in re- cent years (Bengio et al., 2003; Turian et al., 2010; Collobert et al., 2011; Huang et al., 2012). Fol- lowing the idea that the meaning of a word can be determined by ‘the company it keeps’ (Baroni and Zamparelli, 2010), i.e., the words that it co-occurs with, word embedding projects discrete words to a low-dimensional and continuous vector space where co-occurred words are located close to each other. Compared to conventional discrete representations (e.g., the one-hot encoding), word embedding pro- vides more robust representations for words, partic- ulary for those that infrequently appear in the train- ing data. More importantly, the embedding encodes syntactic and semantic content implicitly, so that re- lations among words can be simply computed as the distances among their embeddings, or word vec- tors. A well-known efficient word embedding ap- proach was recently proposed by (Mikolov et al., 2013a), where two log-linear models (CBOW and skip-gram) are proposed to learn the neighboring re- lation of words in context. A following work pro- posed by the same authors introduces some modifi- cations that largely improve the efficiency of model training (Mikolov et al., 2013c). An interesting property of word vectors learned by the log-linear model is that the relations among relevant words seem linear and can be computed by simple vector addition and substraction (Mikolov et al., 2013d). For example, the following relation ap- proximately holds in the word vector space: Paris - France + Rome = Italy. In (Mikolov et al., 2013b), the linear relation is extended to the bilingual sce- nario, where a linear transform is learned to project semantically identical words from one language to another. The authors reported a high accuracy on a bilingual word translation task. Although promising, we argue that both the word embedding and the linear transform are ill-posed, due to the inconsistence …

Content truncated to 500 words