No Word is an Island—A Transformation Weighting Model for Semantic Composition

No Word is an Island—A Transformation Weighting Model for Semantic Composition Corina Dima and Dani¨el de Kok and Neele Witte and Erhard Hinrichs SFB 833 and Seminar f¨ur Sprachwissenschaft, University of T¨ubingen Wilhelmstr. 19, 72074, T¨ubingen, Germany {corina.dima, daniel.de-kok, neele.witte, erhard.hinrichs}@uni-tuebingen.de Abstract Composition models of distributional seman- tics are used to construct phrase representa- tions from the representations of their words. Composition models are typically situated on two ends of a spectrum. They either have a small number of parameters but compose all phrases in the same way, or they perform word- specific compositions at the cost of a far larger number of parameters. In this paper we pro- pose transformation weighting (TransWeight), a composition model that consistently outper- forms existing models on nominal compounds, adjective-noun phrases, and adverb-adjective phrases in English, German, and Dutch. Trans- Weight drastically reduces the number of par- ameters needed compared with the best model in the literature by composing similar words in the same way. 1 Introduction The phrases black car and purple car are very similar—except for their frequency. In a large cor- pus, described in more detail in Section 4.1, the phrase black car occurs 131 times, making it possi- ble to model the whole phrase distributionally. But purple car is far less frequent, with only five occur- rences, and therefore out of reach for distributional models based on co-occurrence counts. Distributionalword representations derived from large, unannotated corpora (Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) capture information about individual words like purple and car and are able to express, in vector space, different types of word similarity (the similarity between color adjectives like black and purple, between car and truck, etc.). Creating phrasal representations, and in par- ticular representing low-frequency phrases like purple car, is a task for composition models of distributional semantics. A composition model is a function f that combines the vectors of individual words, for example, u for black and v for car into a phrase representation p. p is the result of applying the composition function f to the word vectors u, v: p = f(u, v). Our goal is to find the function f that learns how to compose phrase representations by training on a set of phrases. The target representation of the whole phrase, ˜p, is learned directly from the corpus, if needed by concatenating first the word pairs of interest, for example, black car. The function f seeks to maximize the cosine sim- ilarity of the composed representation p and the target representation ˜p, arg max p·˜p ∥p∥2∥˜p∥2 . The contributions of this paper are as follows: • We provide an extensive review and eval- uation of existing composition models. The evaluation is carried out in parallel on three syntactic constructions using English, German, and Dutch treebanks: adjective-noun phrases (black car, schwarz Auto, zwart auto), nom- inal compounds (apple tree, Apfelbaum, appel- boom), and adverb-adjective phrases (very large, sehr groß, zeer groot). These con- structions serve as case studies in …

Content truncated to 500 words