Improving Sparse Word Representations with Distributional Inference for Semantic Composition

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1691–1702, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Improving Sparse Word Representations with Distributional Inference for Semantic Composition Thomas Kober, Julie Weeds, Jeremy Reffin and David Weir TAG laboratory, Department of Informatics, University of Sussex Brighton, BN1 9RH, UK {t.kober, j.e.weeds, j.p.reffin, d.j.weir}@sussex.ac.uk Abstract Distributional models are derived from co- occurrences in a corpus, where only a small proportion of all possible plausible co- occurrences will be observed. This results in a very sparse vector space, requiring a mechanism for inferring missing knowledge. Most methods face this challenge in ways that render the resulting word representa- tions uninterpretable, with the consequence that semantic composition becomes hard to model. In this paper we explore an alter- native which involves explicitly inferring un- observed co-occurrences using the distribu- tional neighbourhood. We show that distribu- tional inference improves sparse word repre- sentations on several word similarity bench- marks and demonstrate that our model is com- petitive with the state-of-the-art for adjective- noun, noun-noun and verb-object composi- tions while being fully interpretable. 1 Introduction The aim of distributional semantics is to derive meaning representations based on observing co- occurrences of words in large text corpora. However not all plausible co-occurrences will be observed in any given corpus, resulting in word representa- tions that only capture a fragment of the meaning of a word. For example the verbs “walking” and “strolling” may occur in many different and pos- sibly disjoint contexts, although both verbs would be equally plausible in numerous cases. This sub- sequently results in incomplete representations for both lexemes. In addition, models based on counting co-occurrences face the general problem of sparsity in a very high-dimensional vector space. The most common approaches to these challenges have in- volved the use of various techniques for dimension- ality reduction (Bullinaria and Levy, 2012; Lapesa and Evert, 2014) or the use of low-dimensional and dense neural word embeddings (Mikolov et al., 2013; Pennington et al., 2014). The common prob- lem in both of these approaches is that composition becomes a black-box process due to the lack of inter- pretability of the representations. Count-based mod- els are therefore a very attractive line of work with regards to a number of important long-term research challenges, most notably the development of an ade- quate model of distributional compositional seman- tics. In this paper we propose the use of distribu- tional inference (DI) to inject unobserved but plau- sible distributional semantic knowledge into the vec- tor space by leveraging the intrinsic structure of the distributional neighbourhood. This results in richer word representations and furthermore mitigates the sparsity effect common in high-dimensional vector spaces, while remaining fully interpretable. Our contributions are as follows: we show that typed and untyped sparse word representations, enriched by distributional inference, lead to performance im- provements on several word similarity benchmarks, and that a higher-order dependency-typed vector space model, based on “Anchored Packed Depen- dency Trees (APTs)” (Weir et al., …

Content truncated to 500 words