Siamese CBOW: Optimizing Word Embeddings for Sentence Representations

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 941–951, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Siamese CBOW: Optimizing Word Embeddings for Sentence Representations Tom Kenter1 Alexey Borisov1, 2 Maarten de Rijke1 tom.kenter@uva.nl alborisov@yandex-team.ru derijke@uva.nl 1 University of Amsterdam, Amsterdam 2 Yandex, Moscow Abstract We present the Siamese Continuous Bag of Words (Siamese CBOW) model, a neural network for efficient estimation of high- quality sentence embeddings. Averaging the embeddings of words in a sentence has proven to be a surprisingly success- ful and efficient way of obtaining sen- tence embeddings. However, word em- beddings trained with the methods cur- rently available are not optimized for the task of sentence representation, and, thus, likely to be suboptimal. Siamese CBOW handles this problem by training word em- beddings directly for the purpose of be- ing averaged. The underlying neural net- work learns word embeddings by predict- ing, from a sentence representation, its surrounding sentences. We show the ro- bustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources. 1 Introduction Word embeddings have proven to be beneficial in a variety of tasks in NLP such as machine translation (Zou et al., 2013), parsing (Chen and Manning, 2014), semantic search (Reinanda et al., 2015; Voskarides et al., 2015), and tracking the meaning of words and concepts over time (Kim et al., 2014; Kenter et al., 2015). It is not evi- dent, however, how word embeddings should be combined to represent larger pieces of text, like sentences, paragraphs or documents. Surprisingly, simply averaging word embeddings of all words in a text has proven to be a strong baseline or feature across a multitude of tasks (Faruqui et al., 2014; Yu et al., 2014; Gershman and Tenenbaum, 2015; Kenter and de Rijke, 2015). Word embeddings, however, are not optimized specifically for representing sentences. In this pa- per we present a model for obtaining word embed- dings that are tailored specifically for the task of averaging them. We do this by directly including a comparison of sentence embeddings—the aver- aged embeddings of the words they contain—in the cost function of our network. Word embeddings are typically trained in a fast and scalable way from unlabeled training data. As the training data is unlabeled, word embeddings are usually not task-specific. Rather, word embed- dings trained on a large training corpus, like the ones from (Collobert and Weston, 2008; Mikolov et al., 2013b) are employed across different tasks (Socher et al., 2012; Kenter and de Rijke, 2015; Hu et al., 2014). These two qualities— (i) being trainable from large quantities of unlabeled data in a reasonable amount of time, and (ii) robust performance across different tasks—are highly de- sirable and allow word embeddings to be used in many large-scale applications. In this work we aim to optimize word embeddings for sentence representations in the same manner. We want to produce general purpose sentence embeddings that should score robustly across multiple test …

Content truncated to 500 words