Multimodal Word Distributions

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1645–1656 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1151 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1645–1656 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1151 Multimodal Word Distributions Ben Athiwaratkun Cornell University pa338@cornell.edu Andrew Gordon Wilson Cornell University andrew@cornell.edu Abstract Word embeddings provide point represen- tations of words containing useful seman- tic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, en- tailment, and rich uncertainty informa- tion. To learn these distributions, we pro- pose an energy-based max-margin objec- tive. We show that the resulting approach captures uniquely expressive semantic in- formation, and outperforms alternatives, such as word2vec skip-grams, and Gaus- sian embeddings, on benchmark datasets such as word similarity and entailment. 1 Introduction To model language, we must represent words. We can imagine representing every word with a binary one-hot vector corresponding to a dictio- nary position. But such a representation contains no valuable semantic information: distances be- tween word vectors represent only differences in alphabetic ordering. Modern approaches, by con- trast, learn to map words with similar meanings to nearby points in a vector space (Mikolov et al., 2013a), from large datasets such as Wikipedia. These learned word embeddings have become ubiquitous in predictive tasks. Vilnis and McCallum (2014) recently proposed an alternative view, where words are represented by a whole probability distribution instead of a de- terministic point vector. Specifically, they model each word by a Gaussian distribution, and learn its mean and covariance matrix from data. This approach generalizes any deterministic point em- bedding, which can be fully captured by the mean vector of the Gaussian distribution. Moreover, the full distribution provides much richer information than point estimates for characterizing words, rep- resenting probability mass and uncertainty across a set of semantics. However, since a Gaussian distribution can have only one mode, the learned uncertainty in this rep- resentation can be overly diffuse for words with multiple distinct meanings (polysemies), in or- der for the model to assign some density to any plausible semantics (Vilnis and McCallum, 2014). Moreover, the mean of the Gaussian can be pulled in many opposing directions, leading to a biased distribution that centers its mass mostly around one meaning while leaving the others not well rep- resented. In this paper, we propose to represent each word with an expressive multimodal distribution, for multiple distinct meanings, entailment, heavy tailed uncertainty, and enhanced interpretability. For example, one mode of the word ‘bank’ could overlap with distributions for words such as ‘fi- nance’ and ‘money’, and another mode could overlap with the distributions for ‘river’ and ‘creek’. It is our contention that such flexibility is critical for both qualitatively learning about the meanings of words, and for optimal performance on many predictive tasks. In particular, we model each word with a mix- ture of Gaussians (Section 3.1). We …

Content truncated to 500 words