context2vec: Learning Generic Context Embedding with Bidirectional LSTM

Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 51–61, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics context2vec: Learning Generic Context Embedding with Bidirectional LSTM Oren Melamud Computer Science Dept. Bar-Ilan University melamuo@cs.biu.ac.il Jacob Goldberger Faculty of Engineering Bar-Ilan University goldbej@eng.biu.ac.il Ido Dagan Computer Science Dept. Bar-Ilan University dagan@cs.biu.ac.il Abstract Context representations are central to vari- ous NLP tasks, such as word sense disam- biguation, named entity recognition, co- reference resolution, and many more. In this work we present a neural model for efficiently learning a generic context em- bedding function from large corpora, us- ing bidirectional LSTM. With a very sim- ple application of our context represen- tations, we manage to surpass or nearly reach state-of-the-art results on sentence completion, lexical substitution and word sense disambiguation tasks, while sub- stantially outperforming the popular con- text representation of averaged word em- beddings. We release our code and pre- trained models, suggesting they could be useful in a wide variety of NLP tasks. 1 Introduction Generic word embeddings capture semantic and syntactic information about individual words in a compact low-dimensional representation. While they are trained to optimize a generic task- independent objective function, word embeddings were found useful in a broad range of NLP tasks, making an overall huge impact in recent years. A major advancement in this field was the in- troduction of highly efficient models, such as word2vec (Mikolov et al., 2013a) and GloVe (Pen- nington et al., 2014), for learning generic word embeddings from very large corpora. Capturing information from such corpora substantially in- creased the value of word embeddings to both un- supervised and semi-supervised NLP tasks. To make inferences regarding a concrete target word instance, good representations of both the target word type and the given context are help- ful. For example, in the sentence “I can’t find [April]”, we need to consider both the target word April and its context “I can’t find [ ]” to infer that April probably refers to a person. This principle applies to various tasks, including word sense dis- ambiguation, co-reference resolution and named entity recognition (NER). Like target words, contexts are commonly rep- resented via word embeddings. In an unsupervised setting, such representations were found useful for measuring context-sensitive similarity (Huang et al., 2012), word sense disambiguation (Chen et al., 2014), word sense induction (K˚ageb¨ack et al., 2015), lexical substitution (Melamud et al., 2015b), sentence completion (Liu et al., 2015) and more. The context representations used in such tasks are commonly just a simple collection of the individual embeddings of the neighboring words in a window around the target word, or a (sometimes weighted) average of these embed- dings. We note that such approaches do not in- clude any mechanism for optimizing the represen- tation of the entire sentential context as a whole. In supervised settings, various NLP systems use labeled data to learn how to consider context word representations in a more optimized task-specific way. This was done in tasks, such …

Content truncated to 500 words