Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1531–1542, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning Jianpeng Cheng University of Oxford Department of Computer Science jianpeng.cheng@stcatz.oxon.org Dimitri Kartsaklis Queen Mary University of London School of Electronic Engineering and Computer Science d.kartsaklis@qmul.ac.uk Abstract Deep compositional models of meaning acting on distributional representations of words in order to produce vectors of larger text constituents are evolving to a pop- ular area of NLP research. We detail a compositional distributional framework based on a rich form of word embeddings that aims at facilitating the interactions between words in the context of a sen- tence. Embeddings and composition lay- ers are jointly learned against a generic objective that enhances the vectors with syntactic information from the surround- ing context. Furthermore, each word is associated with a number of senses, the most plausible of which is selected dy- namically during the composition process. We evaluate the produced vectors qualita- tively and quantitatively with positive re- sults. At the sentence level, the effective- ness of the framework is demonstrated on the MSRPar task, for which we report re- sults within the state-of-the-art range. 1 Introduction Representing the meaning of words by using their distributional behaviour in a large text corpus is a well-established technique in NLP research that has been proved useful in numerous tasks. In a distributional model of meaning, the semantic representation of a word is given as a vector in some high dimensional vector space, obtained ei- ther by explicitly collecting co-occurrence statis- tics of the target word with words belonging to a representative subset of the vocabulary, or by di- rectly optimizing the word vectors against an ob- jective function in some neural-network based ar- chitecture (Collobert and Weston, 2008; Mikolov et al., 2013). Regardless their method of construction, distri- butional models of meaning do not scale up to larger text constituents such as phrases or sen- tences, since the uniqueness of multi-word expres- sions would inevitably lead to data sparsity prob- lems, thus to unreliable vectorial representations. The problem is usually addressed by the provision of a compositional function, the purpose of which is to prepare a vectorial representation for a phrase or sentence by combining the vectors of the words therein. While the nature and complexity of these compositional models may vary, approaches based on deep-learning architectures have been shown to be especially successful in modelling the meaning of sentences for a variety of tasks (Socher et al., 2012; Kalchbrenner et al., 2014). The mutual interaction of distributional word vectors by a means of a compositional model pro- vides many opportunities for interesting research, the majority of which still remains to be explored. One such direction is to investigate in what way lexical ambiguity affects the compositional pro- cess. In fact, recent work has shown that shal- low multi-linear compositional models that explic- itly handle extreme cases of lexical ambiguity in a …

Content truncated to 500 words