Model-Free Context-Aware Word Composition

Proceedings of the 27th International Conference on Computational Linguistics, pages 2834–2845 Santa Fe, New Mexico, USA, August 20-26, 2018. 2834 Model-Free Context-Aware Word Composition Bo An2,3, Xianpei Han1,2, Le Sun1,2 1Beijing Advanced Innovation Center for Language Resources, Beijing, China 2State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences, Beijing, China 3University of Chinese Academy of Sciences, Beijing, China {anbo, xianpei, sunle}@iscas.ac.cn Abstract Word composition is a promising technique for representation learning of large linguistic units (e.g., phrases, sentences and documents). However, most of the current composition models do not take the ambiguity of words and the context outside of a linguistic unit into consideration for learning representations, and consequently suffer from the inaccurate representation of se- mantics. To address this issue, we propose a model-free context-aware word composition model, which employs the latent semantic information as global context for learning representations. The proposed model attempts to resolve the word sense disambiguation and word composition in a unified framework. Extensive evaluation shows consistent improvements over various strong word representation/composition models at different granularities (including word, phrase and sentence), demonstrating the effectiveness of our proposed method. 1 Introduction Recent development in natural language processing (NLP) has seen a rise of continuous representation learning of linguistic units, which has been successfully applied to various tasks such as contextual word similarity (Iacobacci et al., 2015; Huang et al., 2012), machine translation (Devlin et al., 2014), para- phrase detection (Socher et al., 2011a) and sentiment classification (Tang et al., 2015). The continuous representation of linguistic units bring many benefits. For example, we can easily calculate the semantic relatedness between two words ‘dog’ and ‘cat’ using cosine distance of their continuous representa- tions. Specifically, a representation learning method aims at mapping a linguistic unit with k words S = [w1, w2, ..., wk] into a continuous vector in a low dimensional feature space. Currently, most of the representation learning methods focus on learning word embeddings, mostly based on the distributional hypothesis (Harris, 1954), i.e., words in similar contexts tend to have similar meanings. In most cases, a word is represented by summarizing all its contexts in a large text corpus, ei- ther by dimension reduction from co-occurrence matrix (Levy and Goldberg, 2014), or by neural network models (Collobert and Weston, 2008; Mikolov et al., 2013; Huang et al., 2012). Intuitively, a word may express distinct meanings in different contexts, so its representations may be distinct in various linguistic units. For example, the word ‘bank’ should have different representations in ‘commercial bank’ and in ‘river bank’, because they correspond with two distinct senses of ‘bank’: the former with the meaning of ‘financial institution’, and the latter with ‘riverside’ meanings. To address this issue, some multi-prototype word embeddings models are proposed to learn multiple representations for individual words. For instance, MSSG (Neelakantan et al., 2015) utilizes local context to infer the accurate semantic of a word, and the global context is incorporated by TWE (Liu et al., 2015) to learn accurate representations of a …

Content truncated to 500 words