PotTS: The Potsdam Twitter Sentiment Corpus

PotTS: The Potsdam Twitter Sentiment Corpus Uladzimir Sidarenka Applied Computational Linguistics FSP Cognitive Science University of Potsdam Karl-Liebknecht Straße 24-25 14476 Potsdam sidarenk@uni-potsdam.de Abstract In this paper, we introduce a novel comprehensive dataset of 7,992 German tweets, which were manually annotated by two human experts with fine-grained opinion relations. A rich annotation scheme used for this corpus includes such sentiment-relevant elements as opinion spans, their respective sources and targets, emotionally laden terms with their possible contextual negations and modifiers. Various inter-annotator agreement studies, which were carried out at different stages of work on these data (at the initial training phase, upon an adjudication step, and after the final annotation run), reveal that labeling evaluative judgements in microblogs is an inherently difficult task even for professional coders. These difficulties, however, can be alleviated by letting the annotators revise each other’s decisions. Once rechecked, the experts can proceed with the annotation of further messages, staying at a fairly high level of agreement. Keywords: language resources, sentiment analysis, social media, Twitter 1. Introduction As people share more and more personal opinions via social media services, rapidly analyzing these subjec- tive statements in an automatic way becomes a vital necessity for the success of modern social and com- mercial endeavors. This analysis, however, presup- poses the existence of sufficiently big manually an- notated corpora, since these are inevitably required for training new systems and testing existing applica- tions. Although several attempts have already been made to create such datasets for English Twitter (Go et al., 2009; Pak and Paroubek, 2010; Nakov et al., 2013), the number of opinion corpora for this service in less-resourced languages still remains low. In this paper, we try to overcome this limitation by presenting a novel collection of 7,992 German tweets, which were annotated with fine-grained sentiment re- lations by two human experts. With this resource, we not only aim at mitigating the data scarceness prob- lem for German but also attempt to improve the cur- rent state of the art of opinion mining corpora in gen- eral by offering a dataset, which, in contrast to previ- ous distantly supervised and therefore fundamentally noisy works (Go et al., 2009; Barbosa and Feng, 2010; Davidov et al., 2010), was created fully manually with a high level of inter-rater agreement; which, unlike many other hand-labeled sentiment data, is big enough to train and validate various machine-learning tech- niques; and, finally, which not only covers just one aspect of subjective judgements – as it was done, for instance, in the SemEval training set (Nakov et al., 2013), where only expression- and message-level po- larities were annotated – but covers sentiments as a whole, providing precise information about their tex- tual spans as well as the spans of their targets, sources, corresponding polar terms with their possible modi- fiers. We begin our introduction by summarizing related work on opinion corpora for Twitter done so far. Af- ter describing the tracking procedure, which was used to initially collect and pre-select the message data, we …

Content truncated to 500 words