Word Semantic Representations using Bayesian Probabilistic Tensor Factorization

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1522–1531, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Word Semantic Representations using Bayesian Probabilistic Tensor Factorization Jingwei Zhang and Jeremy Salwen Columbia University Computer Science New York, NY 10027, USA {jz2541,jas2312}@columbia.edu Michael Glass and Alfio Gliozzo IBM T.J. Waston Research Yorktown Heights, NY 10598, USA {mrglass,gliozzo}@us.ibm.com Abstract Many forms of word relatedness have been developed, providing different perspec- tives on word similarity. We introduce a Bayesian probabilistic tensor factoriza- tion model for synthesizing a single word vector representation and per-perspective linear transformations from any number of word similarity matrices. The result- ing word vectors, when combined with the per-perspective linear transformation, ap- proximately recreate while also regulariz- ing and generalizing, each word similarity perspective. Our method can combine manually cre- ated semantic resources with neural word embeddings to separate synonyms and antonyms, and is capable of generaliz- ing to words outside the vocabulary of any particular perspective. We evaluated the word embeddings with GRE antonym questions, the result achieves the state-of- the-art performance. 1 Introduction In recent years, vector space models (VSMs) have been proved successful in solving various NLP tasks including named entity recognition, part-of-speech tagging, parsing, semantic role- labeling and answering synonym or analogy ques- tions (Turney et al., 2010; Collobert et al., 2011). Also, VSMs are reported performing well on tasks involving the measurement of word related- ness (Turney et al., 2010). Many existing works are distributional models, based on the Distribu- tional Hypothesis, that words occurring in simi- lar contexts tend to have similar meanings (Har- ris, 1954). The limitation is that word vectors de- veloped from distributional models cannot reveal word relatedness if its information does not lie in word distributions. For instance, they are believed to have difficulty distinguishing antonyms from synonyms, because the distribution of antonymous words are close, since the context of antonymous words are always similar to each other (Moham- mad et al., 2013). Although some research claims that in certain conditions there do exist differ- ences between the contexts of different antony- mous words (Scheible et al., 2013), the differences are subtle enough that it can hardly be detected by such language models, especially for rare words. Another important class of lexical resource for word relatedness is a lexicon, such as Word- Net (Miller, 1995) or Roget’s Thesaurus (Kipfer, 2009). Manually producing or extending lexi- cons is much more labor intensive than generat- ing VSM word vectors using a corpus. Thus, lex- icons are sparse with missing words and multi- word terms as well as missing relationships be- tween words. Considering the synonym / antonym perspective as an example, WordNet answers less than 40% percent of the the GRE antonym ques- tions provided by Mohammad et al. (2008) di- rectly. Moreover, binary entries in lexicons do not indicate the degree of relatedness, such as the de- gree of lexical contrast between happy and sad or happy and depressed. The lack of such informa- …

Content truncated to 500 words