Do Multi-Sense Embeddings Improve Natural Language Understanding?

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1722–1732, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Do Multi-Sense Embeddings Improve Natural Language Understanding? Jiwei Li Computer Science Department Stanford University Stanford, CA 94305, USA jiweil@stanford.edu Dan Jurafsky Computer Science Department Stanford University Stanford, CA 94305, USA jurafsky@stanford.edu Abstract Learning a distinct representation for each sense of an ambiguous word could lead to more powerful and fine-grained mod- els of vector-space representations. Yet while ‘multi-sense’ methods have been proposed and tested on artificial word- similarity tasks, we don’t know if they im- prove real natural language understanding tasks. In this paper we introduce a multi- sense embedding model based on Chinese Restaurant Processes that achieves state of the art performance on matching human word similarity judgments, and propose a pipelined architecture for incorporating multi-sense embeddings into language un- derstanding. We then test the performance of our model on part-of-speech tagging, named entity recognition, sentiment analysis, semantic relation identification and semantic relat- edness, controlling for embedding dimen- sionality. We find that multi-sense embed- dings do improve performance on some tasks (part-of-speech tagging, semantic re- lation identification, semantic relatedness) but not on others (named entity recogni- tion, various forms of sentiment analysis). We discuss how these differences may be caused by the different role of word sense information in each of the tasks. The re- sults highlight the importance of testing embedding models in real applications. 1 Introduction Enriching vector models of word meaning so they can represent multiple word senses per word type seems to offer the potential to improve many language understanding tasks. Most tra- ditional embedding models associate each word type with a single embedding (e.g., Bengio et al. (2006)). Thus the embedding for homonymous words like bank (with senses including ‘sloping land’ and ‘financial institution’) is forced to rep- resent some uneasy central tendency between the various meanings. More fine-grained embeddings that represent more natural regions in semantic space could thus improve language understanding. Early research pointed out that embeddings could model aspects of word sense (Kintsch, 2001) and recent research has proposed a number of models that represent each word type by dif- ferent senses, each sense associated with a sense- specific embedding (Kintsch, 2001; Reisinger and Mooney, 2010; Neelakantan et al., 2014; Huang et al., 2012; Chen et al., 2014; Pina and Johansson, 2014; Wu and Giles, 2015; Liu et al., 2015). Such sense-specific embeddings have shown improved performance on simple artificial tasks like match- ing human word similarity judgments— WS353 (Rubenstein and Goodenough, 1965) or MC30 (Huang et al., 2012). Incorporating multisense word embeddings into general NLP tasks requires a pipelined architec- ture that addresses three major steps: 1. Sense-specific representation learning: learn word sense specific embeddings from a large corpus, either unsupervised or aided by external resources like WordNet. 2. Sense induction: given a text unit (a phrase, sentence, document, etc.), infer word senses for its tokens and associate them with corre- sponding sense-specific embeddings. 3. Representation …

Content truncated to 500 words