Ontology-Aware Token Embeddings for Prepositional Phrase Attachment

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2089–2098 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1191 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2089–2098 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1191 Ontology-Aware Token Embeddings for Prepositional Phrase Attachment Pradeep Dasigi1 Waleed Ammar2 Chris Dyer1,3 Eduard Hovy1 1Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA, USA 2Allen Institute for Artificial Intelligence, Seattle WA, USA 3DeepMind, London, UK pdasigi@cs.cmu.edu, wammar@allenai.org, cdyer@cs.cmu.edu, hovy@cmu.edu Abstract Type-level word embeddings use the same set of parameters to represent all instances of a word regardless of its context, ignor- ing the inherent lexical ambiguity in lan- guage. Instead, we embed semantic con- cepts (or synsets) as defined in WordNet and represent a word token in a partic- ular context by estimating a distribution over relevant semantic concepts. We use the new, context-sensitive embeddings in a model for predicting prepositional phrase (PP) attachments and jointly learn the con- cept embeddings and model parameters. We show that using context-sensitive em- beddings improves the accuracy of the PP attachment model by 5.4% absolute points, which amounts to a 34.4% relative reduction in errors. 1 Introduction Type-level word embeddings map a word type (i.e., a surface form) to a dense vector of real num- bers such that similar word types have similar em- beddings. When pre-trained on a large corpus of unlabeled text, they provide an effective mecha- nism for generalizing statistical models to words which do not appear in the labeled training data for a downstream task. In accordance with standard terminology, we make the following distinction between types and tokens in this paper: By word types, we mean the surface form of the word, whereas by tokens we mean the instantiation of the surface form in a con- text. For example, the same word type ‘pool’ oc- curs as two different tokens in the sentences “He sat by the pool,” and “He played a game of pool.” Most word embedding models define a single vector for each word type. However, a fundamen- tal flaw in this design is their inability to distin- guish between different meanings and abstractions of the same word. In the two sentences shown above, the word ‘pool’ has different meanings, but the same representation is typically used for both of them. Similarly, the fact that ‘pool’ and ‘lake’ are both kinds of water bodies is not explicitly in- corporated in most type-level embeddings. Fur- thermore, it has become a standard practice to tune pre-trained word embeddings as model parameters during training for an NLP task (e.g., Chen and Manning, 2014; Lample et al., 2016), potentially allowing the parameters of a frequent word in the labeled training data to drift away from related but rare words in the embedding space. Previous work partially addresses these prob- lems by estimating concept embeddings in Word- Net (e.g., Rothe and Sch¨utze, 2015), or improv- …

Content truncated to 500 words