Specialising Paragraph Vectors for Text Polarity Detection

Specialising Paragraph Vectors for Text Polarity Detection Fabio Tamburini FICLIT, University of Bologna, Italy fabio.tamburini@unibo.it Abstract This paper presents some experiments for specialising Paragraph Vectors, a new technique for creating text fragment (phrase, sentence, paragraph, text, ...) embedding vectors, for text polarity detection. The first extension regards the injection of polarity information extracted from a polarity lexicon into embeddings and the second extension aimed at inserting word order information into Paragraph Vectors. These two extensions, when training a logistic-regression classifier on the combined embeddings, were able to produce a relevant gain in performance when compared to the standard Paragraph Vector methods proposed by Le and Mikolov (2014). Keywords: Text Polarity Detection, Word Embeddings, Paragraph Vectors, Polarity Lexica. 1. Introduction Distributed word representations, built starting from the well-known Harris (1954) distributional hypothesis often stated as “The degree of semantic similarity between two linguistic expressions A and B is a function of the similarity of the linguistic contexts in which A and B can appear.” and coded as sparse high dimensional vectors, have been play- ing a central role in Natural Language Processing (NLP) for years (Turney and Pantel, 2010; Baroni and Lenci, 2010). More recently, a large set of studies propose to represent words as dense vectors derived by training neural networks for language modelling (Bengio et al., 2003; Collobert and Weston, 2008; Mnih and Hinton, 2009; Mikolov et al., 2013). Such vectors are commonly called “word embed- dings” and have been successfully used in a variety of NLP tasks (e.g. Turian et al. (2010), Collobert et al. (2011), Socher et al. (2013)). These approaches and the cited applications use word- centered embeddings and are not suited, in general, for tasks requiring the classification of an entire text such as topic classification, sentiment analysis, language identifi- cation, etc. Le and Mikolov (2014) proposed an extension of a state-of-the-art word embedding method, namely the widely used word2vec package1 (Mikolov et al., 2013), to create distributed dense representations for phrases, sen- tences, paragraph or even entire texts useful to be used as base vectors for any fragment/text classification task in the NLP domain. Both standard word embeddings and the “Paragraph- Vector” (PV) model from Le and Mikolov produce vectors that do not contain any information about a specific NLP task, but, as showed by all works cited before, they indeed support the production of state-of-the-art classification re- sults in various domains. In this paper we propose to extend PVs in order to specialise them for text polarity classification by injecting specific knowledge connected with this task; we will show in some experiments as the combination of embeddings and exter- nal resources will produce increased performances. The idea of combining word embeddings and external knowl- edge to specialise them for a specific task has been actively explored in the last few years (Wang et al., 2014; Levy and 1https://code.google.com/p/word2vec/ Goldberg, 2014; Tang et al., 2014; Yu and Dredze, 2014; Wang et al., 2015; Pham et al., 2015; Liu et al., 2015), but all these …

Content truncated to 500 words