Vector space semantics with frequency-driven motifs

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 634–643, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Vector space semantics with frequency-driven motifs Shashank Srivastava Carnegie Mellon University Pittsburgh, PA 15217 ssrivastava@cmu.edu Eduard Hovy Carnegie Mellon University Pittsburgh, PA 15217 hovy@cmu.edu Abstract Traditional models of distributional se- mantics suffer from computational issues such as data sparsity for individual lex- emes and complexities of modeling se- mantic composition when dealing with structures larger than single lexical items. In this work, we present a frequency- driven paradigm for robust distributional semantics in terms of semantically cohe- sive lineal constituents, or motifs. The framework subsumes issues such as dif- ferential compositional as well as non- compositional behavior of phrasal con- situents, and circumvents some problems of data sparsity by design. We design a segmentation model to optimally par- tition a sentence into lineal constituents, which can be used to define distributional contexts that are less noisy, semantically more interpretable, and linguistically dis- ambiguated. Hellinger PCA embeddings learnt using the framework show competi- tive results on empirical tasks. 1 Introduction Meaning in language is a confluence of experien- tially acquired semantics of words or multi-word phrases, and their semantic composition to create new meanings. For instance, successfully inter- preting a sentence such as The old senator kicked the bucket. requires the knowledge that the semantic conno- tations of ‘kicking the bucket’ as a unit are the same as those for ‘dying’. Short of explicit su- pervision, such semantic mappings must be in- ferred by a new language speaker through induc- tive mechanisms operating on observed linguis- tic usage. This perspective of acquired meaning aligns with the ‘meaning is usage’ adage, conso- nant with Wittgenstein’s view of semantics. At the same time, the ability to adaptively commu- nicate elaborate meanings can only be conciled through Frege’s principle of compositionality, i.e., meanings of larger linguistic constructs can be derived from the meanings of individual compo- nents, modulated by their syntactic interrelations. Indeed, most linguistic usage appears composi- tional. This is supported by the fact even with very limited vocabulary, children and non-native speakers can often communicate surprisingly ef- fectively. It can be argued that to be sustainable, induc- tive aspects of meaning must be recurrent enough to be learnable by new users. That is, a non- compositional phrase such as ‘kick the bucket’ is likely to persist in common parlance only if it is frequently used with its associated semantic map- ping. If a usage-driven meaning of a motif is not recurrent enough, learning this mapping is inef- ficient in two ways. First, the sparseness of ob- servations would severely limit accurate inductive acquisition by new observers. Second, the value of learning a very infrequent semantic mapping is likely marginal. This motivates the need for a frequency-driven view of lexical semantics. In particular, such a perspective can be especially advantageous for distributional semantics for rea- sons we outline below. Distributional semantic models (DSMs) that represent words as distributions …

Content truncated to 500 words