#TagSpace: Semantic Embeddings from Hashtags

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1822–1827, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics #TAGSPACE: Semantic Embeddings from Hashtags Jason Weston Facebook AI Research jase@fb.com Sumit Chopra Facebook AI Research spchopra@fb.com Keith Adams Facebook AI Research kma@fb.com Abstract We describe a convolutional neural net- work that learns feature representations for short textual posts using hashtags as a su- pervised signal. The proposed approach is trained on up to 5.5 billion words predict- ing 100,000 possible hashtags. As well as strong performance on the hashtag predic- tion task itself, we show that its learned representation of text (ignoring the hash- tag labels) is useful for other tasks as well. To that end, we present results on a docu- ment recommendation task, where it also outperforms a number of baselines. 1 Introduction Hashtags (single tokens often composed of nat- ural language n-grams or abbreviations, prefixed with the character ‘#’) are ubiquitous on social networking services, particularly in short textual documents (a.k.a. posts). Authors use hashtags to diverse ends, many of which can be seen as labels for classical NLP tasks: disambiguation (chips #futurism vs. chips #junkfood); identi- fication of named entities (#sf49ers); sentiment (#dislike); and topic annotation (#yoga). Hashtag prediction is the task of mapping text to its accompanying hashtags. In this work we pro- pose a novel model for hashtag prediction, and show that this task is also a useful surrogate for learning good representations of text. Latent representations, or embeddings, are vec- torial representations of words or documents, tra- ditionally learned in an unsupervised manner over large corpora. For example LSA (Deerwester et al., 1990) and its variants, and more recent neural- network inspired methods like those of Bengio et al. (2006), Collobert et al. (2011) and word2vec (Mikolov et al., 2013) learn word embeddings. In the word embedding paradigm, each word is rep- resented as a vector in Rn, where n is a hyper- parameter that controls capacity. The embeddings of words comprising a text are combined using a model-dependent, possibly learned function, pro- ducing a point in the same embedding space. A similarity measure (for example, inner product) gauges the pairwise relevance of points in the em- bedding space. Unsupervised word embedding methods train with a reconstruction objective in which the em- beddings are used to predict the original text. For example, word2vec tries to predict all the words in the document, given the embeddings of sur- rounding words. We argue that hashtag predic- tion provides a more direct form of supervision: the tags are a labeling by the author of the salient aspects of the text. Hence, predicting them may provide stronger semantic guidance than unsuper- vised learning alone. The abundance of hashtags in real posts provides a huge labeled dataset for learning potentially sophisticated models. In this work we develop a convolutional net- work for large scale ranking tasks, and apply it to hashtag prediction. Our model represents both words and the entire textual post …

Content truncated to 500 words