A Markovian approach to distributional semantics with application to semantic compositionality

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1447–1456, Dublin, Ireland, August 23-29 2014. A Markovian approach to distributional semantics with application to semantic compositionality ´Edouard Grave EECS Department UC Berkeley grave@berkeley.edu Guillaume Obozinski LIGM – Universit´e Paris-Est ´Ecole des Ponts – ParisTech guillaume.obozinski @imagine.enpc.fr Francis Bach Inria – Sierra project-team ´Ecole Normale Sup´erieure francis.bach@ens.fr Abstract In this article, we describe a new approach to distributional semantics. This approach relies on a generative model of sentences with latent variables, which takes the syntax into account by using syntactic dependency trees. Words are then represented as posterior distributions over those latent classes, and the model allows to naturally obtain in-context and out-of-context word representations, which are comparable. We train our model on a large corpus and demonstrate the compositionality capabilities of our approach on different datasets. 1 Introduction It is often considered that words appearing in similar contexts tend to have similar meaning (Harris, 1954). This idea, known as the distributional hypothesis was famously summarized by Firth (1957) as follow: “you shall know a word by the company it keeps.” The distributional hypothesis has been applied in computational linguistics in order to automatically build word representations that capture their meaning. For example, simple distributional information about words, such as co-occurence counts, can be extracted from a large text corpus, and used to build a vectorial representation of words (Lund and Burgess, 1996; Landauer and Dumais, 1997). According to the distributional hypothesis, two words having similar vectorial representations must have similar meanings. It is thus possible and easy to compare words using their vectorial representations. In natural languages, sentences are formed by the composition of simpler elements: words. It is thus reasonable to assume that the meaning of a sentence is determined by combining the meanings of its parts and the syntactic relations between them. This principle, often attributed to the German logician Frege, is known as semantic compositionality. Recently, researchers in computational linguistics started to investigate how the principle of compositionality could be applied to distributional models of semantics (Clark and Pulman, 2007; Mitchell and Lapata, 2008). Given the representations of individual words, such as federal and agency, is it possible to combine them in order to obtain a representation capturing the meaning of the noun phrase federal agency? Most approaches to distributional semantics represent words as vectors in a high-dimensional space and use linear algebra operations to combine individual word representations in order to obtain represen- tations for complex units. In this article, we propose a probabilistic approach to distributional semantics. This approach is based on the generative model of sentences with latent variables, which was introduced by Grave et al. (2013). We make the following contributions: • Given the model introduced by Grave et al. (2013), we describe how in-context and ouf-of-context words can be represented by posterior distributions over latent variables (section 4). • We evaluate out-of-context representations on human similarity judgements prediction tasks and determine what kind of semantic relations are …

Content truncated to 500 words