Learning Composition Models for Phrase Embeddings

Learning Composition Models for Phrase Embeddings Mo Yu Machine Intelligence & Translation Lab Harbin Institute of Technology Harbin, China gflfof@gmail.com Mark Dredze Human Language Technology Center of Excellence Center for Language and Speech Processing Johns Hopkins University Baltimore, MD, 21218 mdredze@cs.jhu.edu Abstract Lexical embeddings can serve as useful rep- resentations for words for a variety of NLP tasks, but learning embeddings for phrases can be challenging. While separate embeddings are learned for each word, this is infeasible for every phrase. We construct phrase em- beddings by learning how to compose word embeddings using features that capture phrase structure and context. We propose efficient unsupervised and task-specific learning objec- tives that scale our model to large datasets. We demonstrate improvements on both language modeling and several phrase semantic simi- larity tasks with various phrase lengths. We make the implementation of our model and the datasets available for general use. 1 Introduction Word embeddings learned by neural language mod- els (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013b) have been success- fully applied to a range of tasks, including syn- tax (Collobert and Weston, 2008; Turian et al., 2010; Collobert, 2011) and semantics (Huang et al., 2012; Socher et al., 2013b; Hermann et al., 2014). However, phrases are critical for capturing lexical meaning for many tasks. For example, Collobert and Weston (2008) showed that word embeddings yielded state-of-the-art systems on word-oriented tasks (POS, NER) but performance on phrase ori- ented tasks, such as SRL, lags behind. We propose a new method for compositional se- mantics that learns to compose word embeddings into phrases. In contrast to a common approach to phrase embeddings that uses pre-defined compo- sition operators (Mitchell and Lapata, 2008), e.g., component-wise sum/multiplication, we learn com- position functions that rely on phrase structure and context. Other work on learning compositions relies on matrices/tensors as transformations (Socher et al., 2011; Socher et al., 2013a; Hermann and Blun- som, 2013; Baroni and Zamparelli, 2010; Socher et al., 2012; Grefenstette et al., 2013). However, this work suffers from two primary disadvantages. First, these methods have high computational complexity for dense embeddings: O(d2) or O(d3) for compos- ing every two components with d dimensions. The high computational complexity restricts these meth- ods to use very low-dimensional embeddings (25 or 50). While low-dimensional embeddings perform well for syntax (Socher et al., 2013a) and sentiment (Socher et al., 2013b) tasks, they do poorly on se- mantic tasks. Second, because of the complexity, they use supervised training with small task-specific datasets. An exception is the unsupervised objec- tive of recursive auto-encoders (Socher et al., 2011). Yet this work cannot utilize contextual features of phrases and still poses scaling challenges. In this work we propose a novel compositional transformation called the Feature-rich Composi- tional Transformation (FCT) model. phrases from their word components.FCT produces In contrast to previous work, our approach to phrase composi- tion can efficiently utilize high dimensional embed- dings (e.g. d = 200) with an unsupervised objective, both of which are critical …

Content truncated to 500 words