Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection

Proceedings of NAACL-HLT 2018, pages 485–495 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection Haw-Shiuan Chang1, ZiYun Wang2, Luke Vilnis1, Andrew McCallum1 1University of Massachusetts, Amherst, USA 2Tsinghua University, Beijing, China hschang@cs.umass.edu, wang-zy14@mails.tsinghua.edu.cn, {luke, mccallum}@cs.umass.edu Abstract Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, coref- erence, relation extraction, and question an- swering. Supervised learning from labeled hypernym sources, such as WordNet, limits the coverage of these models, which can be addressed by learning hypernyms from un- labeled text. Existing unsupervised meth- ods either do not scale to large vocabularies or yield unacceptably poor accuracy. This paper introduces distributional inclusion vec- tor embedding (DIVE), a simple-to-implement unsupervised method of hypernym discov- ery via per-word non-negative vector embed- dings which preserve the inclusion property of word contexts in a low-dimensional and interpretable space. In experimental evalua- tions more comprehensive than any previous literature of which we are aware—evaluating on 11 datasets using multiple existing as well as newly proposed scoring functions—we find that our method provides up to double the pre- cision of previous unsupervised embeddings, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art results. 1 Introduction Numerous applications benefit from compactly representing context distributions, which assign meaning to objects under the rubric of distribu- tional semantics. In natural language process- ing, distributional semantics has long been used to assign meanings to words (that is, to lex- emes in the dictionary, not individual instances of word tokens). The meaning of a word in the distributional sense is often taken to be the set of textual contexts (nearby tokens) in which that word appears, represented as a large sparse bag of words (SBOW). Without any supervision, Word2Vec (Mikolov et al., 2013), among other approaches based on matrix factorization (Levy et al., 2015a), successfully compress the SBOW into a much lower dimensional embedding space, increasing the scalability and applicability of the embeddings while preserving (or even improving) the correlation of geometric embedding similari- ties with human word similarity judgments. While embedding models have achieved im- pressive results, context distributions capture more semantic information than just word similarity. The distributional inclusion hypothesis (DIH) (Weeds and Weir, 2003; Geffet and Dagan, 2005; Cimiano et al., 2005) posits that the context set of a word tends to be a subset of the contexts of its hy- pernyms. For a concrete example, most adjectives that can be applied to poodle can also be applied to dog, because dog is a hypernym of poodle (e.g. both can be obedient). However, the converse is not necessarily true — a dog can be straight-haired but a poodle cannot. Therefore, dog tends to have a broader context set than poodle. Many asymmet- ric scoring functions comparing SBOW features based on DIH have been developed for hypernymy detection (Weeds and Weir, 2003; Geffet and Da- gan, 2005; Shwartz et al., …

Content truncated to 500 words