A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment

A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment Jing Wang∗ Mohit Bansal† Kevin Gimpel† Brian D. Ziebart∗ Clement T. Yu∗ ∗University of Illinois at Chicago, Chicago, IL, 60607, USA {jwang69,bziebart,cyu}@uic.edu †Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {mbansal,kgimpel}@ttic.edu Abstract Word sense induction (WSI) seeks to automat- ically discover the senses of a word in a cor- pus via unsupervised methods. We propose a sense-topic model for WSI, which treats sense and topic as two separate latent vari- ables to be inferred jointly. Topics are in- formed by the entire document, while senses are informed by the local context surrounding the ambiguous word. We also discuss unsu- pervised ways of enriching the original cor- pus in order to improve model performance, including using neural word embeddings and external corpora to expand the context of each data instance. We demonstrate significant im- provements over the previous state-of-the-art, achieving the best results reported to date on the SemEval-2013 WSI task. 1 Introduction Word sense induction (WSI) is the task of automat- ically discovering all senses of an ambiguous word in a corpus. The inputs to WSI are instances of the ambiguous word with its surrounding context. The output is a grouping of these instances into clusters corresponding to the induced senses. WSI is gen- erally conducted as an unsupervised learning task, relying on the assumption that the surrounding con- text of a word indicates its meaning. Most previous work assumed that each instance is best labeled with a single sense, and therefore, that each instance be- longs to exactly one sense cluster. However, recent work (Erk and McCarthy, 2009; Jurgens, 2013) has shown that more than one sense can be used to inter- pret certain instances, due to context ambiguity and sense relatedness. To handle these characteristics of WSI (unsuper- vised, senses represented by token clusters, multiple senses per instance), we consider approaches based on topic models. A topic model is an unsupervised method that discovers the semantic topics underly- ing a collection of documents. The most popular is latent Dirichlet allocation (LDA; Blei et al., 2003), in which each topic is represented as a multinomial distribution over words, and each document is rep- resented as a multinomial distribution over topics. One approach would be to run LDA on the in- stances for an ambiguous word, then simply inter- pret topics as induced senses (Brody and Lapata, 2009). However, while sense and topic are related, they are distinct linguistic phenomena. Topics are assigned to entire documents and are expressed by all word tokens, while senses relate to a single am- biguous word and are expressed through the local context of that word. One possible approach would be to only keep the local context of each ambigu- ous word, discarding the global context. However, the topical information contained in the broader con- text, though it may not determine the sense directly, might still be useful for narrowing down the likely senses of the ambiguous word. Consider the ambiguous word …

Content truncated to 500 words