A Walk-Based Semantically Enriched Tree Kernel Over Distributed Word Representations

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1411–1416, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics A Walk-based Semantically Enriched Tree Kernel Over Distributed Word Representations Shashank Srivastava1 Dirk Hovy2 Eduard Hovy1 (1) Carnegie Mellon University, Pittsburgh (2) Center for Language Technology, University of Copenhagen, Denmark {ssrivastava,hovy}@cmu.edu, mail@dirkhovy.com Abstract In this paper, we propose a walk-based graph kernel that generalizes the notion of tree- kernels to continuous spaces. Our proposed approach subsumes a general framework for word-similarity, and in particular, provides a flexible way to incorporate distributed repre- sentations. Using vector representations, such an approach captures both distributional se- mantic similarities among words as well as the structural relations between them (encoded as the structure of the parse tree). We show an ef- ficient formulation to compute this kernel us- ing simple matrix operations. We present our results on three diverse NLP tasks, showing state-of-the-art results. 1 Introduction Capturing semantic similarity between sentences is a fundamental issue in NLP, with applications in a wide range of tasks. Previously, tree kernels based on common substructures have been used to model similarity between parse trees (Collins and Duffy, 2002; Moschitti, 2004; Moschitti, 2006b). These kernels encode a high number of latent syntactic features within a concise representation, and com- pute the similarity between two parse trees based on the matching of node-labels (words, POS tags, etc.), as well as the overlap of tree structures. While this is sufficient to capture syntactic similarity, it does not capture semantic similarity very well, even when using discrete semantic types as node labels. This constrains the utility of many traditional tree kernels in two ways: i) two sentences that are syntactically identical, but have no semantic similarity can receive a high matching score (see Table 1, top) while ii) two sentences with only local syntactic overlap, but high semantic similarity can receive low scores (see Table 1, bottom). tree pairs semantic syntactic score � ✓ high ✓ � low love we toys crush they puppies kissed she cat gave she her kiss a friend feline green little her Table 1: Traditional tree kernels do not capture se- mantic similarity In contrast, distributional vector representations of words have been successful in capturing fine- grained semantics, but lack syntactic knowledge. Resources such as Wordnet, dictionaries and on- tologies that encode different semantic perspectives can also provide additional knowledge infusion. In this paper, we describe a generic walk-based graph kernel for dependency parse trees that sub- sumes general notions of word-similarity, while focusing on vector representations of words to capture lexical semantics. Through a convolutional framework, our approach takes into account the distributional semantic similarities between words in a sentence as well as the structure of the parse tree. Our main contributions are: 1. We present a new graph kernel for NLP that ex- tends to distributed word representations, and diverse word similarity measures. 2. Our proposed approach provides a flexible framework for incorporating both syntax and semantics of sentence …

Content truncated to 500 words