Documents and Dependencies: an Exploration of Vector Space Models for Semantic Composition

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 84–93, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Documents and Dependencies: an Exploration of Vector Space Models for Semantic Composition Alona Fyshe, Partha Talukdar, Brian Murphy and Tom Mitchell Machine Learning Department & Center for the Neural Basis of Cognition Carnegie Mellon University, Pittsburgh {afyshe|ppt|bmurphy|tom.mitchell}@cs.cmu.edu Abstract In most previous research on distribu- tional semantics, Vector Space Models (VSMs) of words are built either from topical information (e.g., documents in which a word is present), or from syntac- tic/semantic types of words (e.g., depen- dency parse links of a word in sentences), but not both. In this paper, we explore the utility of combining these two representa- tions to build VSM for the task of seman- tic composition of adjective-noun phrases. Through extensive experiments on bench- mark datasets, we find that even though a type-based VSM is effective for seman- tic composition, it is often outperformed by a VSM built using a combination of topic- and type-based statistics. We also introduce a new evaluation task wherein we predict the composed vector represen- tation of a phrase from the brain activity of a human subject reading that phrase. We exploit a large syntactically parsed corpus of 16 billion tokens to build our VSMs, with vectors for both phrases and words, and make them publicly available. 1 Introduction Vector space models (VSMs) of word semantics use large collections of text to represent word meanings. Each word vector is composed of fea- tures, where features can be derived from global corpus co-occurrence patterns (e.g. how often a word appears in each document), or local corpus co-occurrence patterns patterns (e.g. how often two words appear together in the same sentence, or are linked together in dependency parsed sen- tences). These two feature types represent dif- ferent aspects of word meaning (Murphy et al., 2012c), and can be compared with the paradig- matic/syntagmatic distinction (Sahlgren, 2006). Global patterns give a more topic-based mean- ing (e.g. judge might appear in documents also containing court and verdict). Certain local pat- terns give a more type-based meaning (e.g. the noun judge might be modified by the adjective harsh, or be the subject of decide, as would related and substitutable words such as referee or con- ductor). Global patterns have been used in Latent Semantic Analysis (Landauer and Dumais, 1997) and LDA Topic models (Blei et al., 2003). Local patterns based on word co-occurrence in a fixed width window were used in Hyperspace Analogue to Language (Lund and Burgess, 1996). Subse- quent models added increasing linguistic sophisti- cation, up to full syntactic and dependency parses (Lin, 1998; Pad´o and Lapata, 2007; Baroni and Lenci, 2010). In this paper we systematically explore the util- ity of a global, topic-based VSM built from what we call Document features, and a local, type-based VSM built from Dependency features. Our Doc- ument VSM represents each word w by a vector where each feature is a specific document, and the feature …

Content truncated to 500 words