How Topic Biases Your Results? A Case Study of Sentiment Analysis and Irony Detection in Italian

Proceedings of Recent Advances in Natural Language Processing, pages 41–47, Hissar, Bulgaria, Sep 7–9 2015. How Topic Biases Your Results? A Case Study of Sentiment Analysis and Irony Detection in Italian Francesco Barbieri, Francesco Ronzano, Horacio Saggion Universitat Pompeu Fabra, DTIC, Barcelona, Spain name.surname@upf.edu Abstract In this paper we present our approach to automatically identify the subjectivity, po- larity and irony of Italian Tweets. Our system which reaches and outperforms the state of the art in Italian is well adapted for different domains since it uses abstract word features instead of bag of words. We also present experiments carried out to study how Italian Sentiment Analysis sys- tems react to domain changes. We show that bag of words approaches commonly used in Sentiment Analysis do not adapt well to domain changes. 1 Introduction The automatic identification of sentiments and opinions expressed by users online is a signif- icant and challenging research trend. The task becomes even more difficult when dealing with short and informal texts like Tweets and other microblog texts. Sentiment Analysis of Tweets has been already investigated by several research studies (Jansen et al., 2009; Barbosa and Feng, 2010). Moreover, during the last few years, many evaluation campaigns have been organised to dis- cuss and compare Sentiment Analysis systems tai- lored to Tweets. Among these campaigns, since 2013, in the context of SemEval (Nakov et al., 2013), several tasks targeting Sentiment Analysis of English Short Texts took place. In 2014, SEN- TIPOLC (Basile et al., 2014), the SENTIment PO- Larity Classification Task of Italian Tweets, was organized in the context of EVALITA 2014, the fourth evaluation campaign of Natural Language Processing and Speech tools for Italian. SEN- TIPOLC distributed a dataset of Italian Tweets an- notated with respect to subjectivity, polarity and irony. This dataset enabled training, evaluation and comparison of the systems that participated to the three tasks of SENTIPOLC, respectively deal- ing with Subjectivity, Polarity and Irony detection. In the Subjectivity task participants were asked to recognise whether a Tweet is objective or subjec- tive, in the Polarity Task they were asked to clas- sify Tweets as positive or negative, and finally, in the Irony Task to detect whether the content of a Tweet is ironic. The following Tweets include an example of each SENTIPOLC class: • Objective Tweet: RT @user: Fine primo tempo: #Fiorentina- Juve 0-2 (Tevez, Pogba). Quali sono i vostri commenti sui primi 45 minuti?#ForzaJuve (RT @user: First half: #FiorentinaJuve 0-2 (Tevez, Pogba). What are your comments on the first 45 minutes? #GOJUVE) • Subjective / Positive / Non-Ironic Tweet: io vorrei andare a votare, ma non penso sia il momento di perder altro tempo e soprattutto denaro.Un governo Monti potrebbe andare. E x voi? (I would like to vote, but I do not think it is the moment to waste time and money. Monti’s government might work. What do you think?) • Subjective / Negative / Ironic Tweet: Brunetta sostiene di tornare a fare l’economista, Mario Monti terrorizzato progetta di mollare tutto …

Content truncated to 500 words