Decoding Sentiment from Distributed Representations of Sentences

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 22–32, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Decoding Sentiment from Distributed Representations of Sentences Edoardo Maria Ponti ep490@cam.ac.uk Ivan Vuli´c iv250@cam.ac.uk Language Technology Lab, University of Cambridge Anna Korhonen alk23@cam.ac.uk Abstract Distributed representations of sentences have been developed recently to represent their meaning as real-valued vectors. How- ever, it is not clear how much information such representations retain about the po- larity of sentences. To study this question, we decode sentiment from unsupervised sentence representations learned with dif- ferent architectures (sensitive to the order of words, the order of sentences, or none) in 9 typologically diverse languages. Senti- ment results from the (recursive) composi- tion of lexical items and grammatical strate- gies such as negation and concession. The results are manifold: we show that there is no ‘one-size-fits-all’ representation ar- chitecture outperforming the others across the board. Rather, the top-ranking archi- tectures depend on the language and data at hand. Moreover, we find that in several cases the additive composition model based on skip-gram word vectors may surpass su- pervised state-of-art architectures such as bidirectional LSTMs. Finally, we provide a possible explanation of the observed varia- tion based on the type of negative construc- tions in each language. 1 Introduction Distributed representations of sentences are usu- ally acquired in an unsupervised fashion from raw texts. Those inferred from different algorithms are prone to grasp parts of their meaning and disregard others. Representations have been evaluated thor- oughly, both intrinsically (interpretation through distance measures) and extrinsically (performance on downstream tasks). Moreover, several methods have been considered, based on both the compo- sition of word embeddings (Milajevs et al., 2014; Marelli et al., 2014; Sultan et al., 2015) and direct generation (Hill et al., 2016). The evaluation was focused solely on English, and it rarely concerned other languages (Adi et al., 2017; Conneau et al., 2017). As a consequence, many ‘core’ methods to learn distributed sentence representations are largely under-explored in a variety of typologically diverse languages, and still lack a demonstration of their usefulness in actual downstream tasks. In this work, we study how well distributed sen- tence representations capture the polarity of a sen- tence. To this end, we choose the Sentiment Anal- ysis task as an extrinsic evaluation protocol: it di- rectly detects the polarity of a text, where polarity is defined as the attitude of the speaker with respect to the whole content of the string or one of the enti- ties mentioned therein. This attitude is measured quantitatively on a scale spanning from negative to positive with arbitrary granularity. As such, po- larity consists in a crucial part of the meaning of a sentence, which should not be lost. The polarity of a sentence depends heavily on a complex interaction between lexical items endowed with an intrinsic polarity, and morphosyntactic con- structions altering polarity, most notably negation and concession. The interaction is deemed to be recursive, hence some approaches …

Content truncated to 500 words