WordRank: Learning Word Embeddings via Robust Ranking

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 658–668, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics WordRank: Learning Word Embeddings via Robust Ranking Shihao Ji Parallel Computing Lab, Intel shihao.ji@intel.com Hyokun Yun Amazon yunhyoku@amazon.com Pinar Yanardag Purdue University ypinar@purdue.edu Shin Matsushima University of Tokyo shin matsushima@mist. i.u-tokyo.ac.jp S. V. N. Vishwanathan Univ. of California, Santa Cruz vishy@ucsc.edu Abstract Embedding words in a vector space has gained a lot of attention in recent years. While state- of-the-art methods provide efficient computa- tion of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a rank- ing problem due to the ranking nature of the evaluation metrics. Then, based on this in- sight, we propose a novel framework Wor- dRank that efficiently estimates word repre- sentations via robust ranking, in which the attention mechanism and robustness to noise are readily achieved via the DCG-like rank- ing losses. The performance of WordRank is measured in word similarity and word anal- ogy benchmarks, and the results are com- pared to the state-of-the-art word embedding techniques. Our algorithm is very competi- tive to the state-of-the- arts on large corpora, while outperforms them by a significant mar- gin when the training set is limited (i.e., sparse and noisy). With 17 million tokens, WordRank performs almost as well as existing methods using 7.2 billion tokens on a popular word similarity benchmark. Our multi-node dis- tributed implementation of WordRank is pub- licly available for general usage. 1 Introduction Embedding words into a vector space, such that se- mantic and syntactic regularities between words are preserved, is an important sub-task for many appli- cations of natural language processing. Mikolov et al. (2013a) generated considerable excitement in the machine learning and natural language processing communities by introducing a neural network based model, which they call word2vec. It was shown that word2vec produces state-of-the-art performance on both word similarity as well as word analogy tasks. The word similarity task is to retrieve words that are similar to a given word. On the other hand, word analogy requires answering queries of the form a:b;c:?, where a, b, and c are words from the vocab- ulary, and the answer to the query must be semanti- cally related to c in the same way as b is related to a. This is best illustrated with a concrete example: Given the query king:queen;man:? we expect the model to output woman. The impressive performance of word2vec led to a flurry of papers, which tried to explain and im- prove the performance of word2vec both theoreti- cally (Arora et al., 2015) and empirically (Levy and Goldberg, 2014). One interpretation of word2vec is that it is approximately maximizing the positive pointwise mutual information (PMI), and Levy and Goldberg (2014) showed that directly optimizing this gives good results. On the other hand, Penning- ton et al. (2014) showed performance comparable to word2vec by using a …

Content truncated to 500 words