Query Expansion with Locally-Trained Word Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 367–377, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Query Expansion with Locally-Trained Word Embeddings Fernando Diaz Microsoft fdiaz@microsoft.com Bhaskar Mitra Microsoft bmitra@microsoft.com Nick Craswell Microsoft nickr@microsoft.com Abstract Continuous space word embeddings have received a great deal of atten- tion in the natural language processing and machine learning communities for their ability to model term similarity and other relationships. We study the use of term relatedness in the context of query expansion for ad hoc informa- tion retrieval. We demonstrate that word embeddings such as word2vec and GloVe, when trained globally, under- perform corpus and query specific em- beddings for retrieval tasks. These re- sults suggest that other tasks benefit- ing from global embeddings may also benefit from local embeddings. 1 Introduction Continuous space embeddings such as word2vec (Mikolov et al., 2013b) or GloVe (Pennington et al., 2014a) project terms in a vocabulary to a dense, lower dimensional space. Recent results in the natural lan- guage processing community demonstrate the effectiveness of these methods for analogy and word similarity tasks. In general, these approaches provide global representations of words; each word has a fixed representation, regardless of any discourse context. While a global representation provides some advan- tages, language use can vary dramatically by topic. For example, ambiguous terms can eas- ily be disambiguated given local information in immediately surrounding words (Harris, 1954; Yarowsky, 1993). The window-based training of word2vec style algorithms exploits this distributional property. A global word embedding, even when trained using local windows, risks captur- ing only coarse representations of those top- ics dominant in the corpus. While a par- ticular embedding may be appropriate for a specific word within a sentence-length con- text globally, it may be entirely inappropri- ate within a specific topic. Gale et al. re- fer to this as the ‘one sense per discourse’ property (Gale et al., 1992). Previous work by Yarowsky demonstrates that this property can be successfully combined with informa- tion from nearby terms for word sense dis- ambiguation (Yarowsky, 1995). Our work ex- tends this approach to word2vec-style training in the context word similarity. For many tasks that require topic-specific linguistic analysis, we argue that topic-specific representations should outperform global rep- resentations. Indeed, it is difficult to imagine a natural language processing task that would not benefit from an understanding of the local topical structure. Our work focuses on a query expansion, an information retrieval task where we can study different lexical similarity meth- ods with an extrinsic evaluation metric (i.e. retrieval metrics). Recent work has demon- strated that similarity based on global word embeddings can be used to outperform clas- sic pseudo-relevance feedback techniques (Sor- doni et al., 2014; al Masri et al., 2016). We propose that embeddings be learned on topically-constrained corpora, instead of large topically-unconstrained corpora. In a retrieval scenario, this amounts to retraining an em- bedding on documents related to the topic of the query. We present local embeddings which …

Content truncated to 500 words