SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 470–478, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter Aniruddha Ghosh Guofu Li University College Dublin, Ireland. University College Dublin, Ireland. arghyaonline@gmail.com li.guofu.l@gmail.com Tony Veale University College Dublin, Ireland. Paolo Rosso Tony.Veale@UCD.ie Universitat Politècnica de València, Spain. Ekaterina Shutova prosso@dsic.upv.es University of Cambridge. Ekaterina.Shutova@cl.cam.ac.uk Antonio Reyes John Barnden Instituto Superior de Intérpretes y Traductores University of Birmingham, UK Mexico J.A.Barnden@cs.bham.ac.uk antonioreyes@isit.edu.mx Abstract This report summarizes the objectives and evaluation of the SemEval 2015 task on the sentiment analysis of figurative language on Twitter (Task 11). This is the first sentiment analysis task wholly dedicated to analyzing figurative language on Twitter. Specifically, three broad classes of figurative language are considered: irony, sarcasm and metaphor. Gold standard sets of 8000 training tweets and 4000 test tweets were annotated using workers on the crowdsourcing platform CrowdFlower. Participating systems were required to provide a fine-grained sentiment score on an 11-point scale (-5 to +5, including 0 for neutral intent) for each tweet, and systems were evaluated against the gold standard using both a Cosine- similarity and a Mean-Squared-Error measure. 1 Introduction The limitations on text length imposed by micro- blogging services such as Twitter do nothing to dampen our willingness to use language creatively. Indeed, such limitations further incentivize the use of creative devices such as metaphor and irony, as such devices allow strongly-felt sentiments to be expressed effectively, memorably and concisely. Nonetheless, creative language can pose certain challenges for NLP tools that do not take account of how words can be used playfully and in original ways. In the case of language using figurative devices such as irony, sarcasm or metaphor – when literal meanings are discounted and secondary or extended meanings are intentionally profiled – the affective polarity of the literal meaning may differ significantly from that of the intended figurative meaning. Nowhere is this effect more pronounced than in ironical language, which delights in using affirmative language to convey critical meanings. Metaphor, irony and sarcasm can each sculpt the affect of an utterance in complex ways, and each tests the limits of conventional techniques for the sentiment analysis of supposedly literal texts. Figurative language thus poses an especially significant challenge to sentiment analysis systems, as standard approaches anchored in the dictionary- defined affect of individual words and phrases are often shown to be inadequate in the face of indirect figurative meanings. It would be convenient if such language were rare and confined to specific genres of text, such as poetry and literature. Yet the reality is that figurative language is pervasive in almost any genre of text, and is especially commonplace on the texts of the Web and on social media platforms such as Twitter. Figurative language often draws attention to itself as a creative artifact, but is just as likely to be viewed as part of the general fabric of human communication. In any case, Web …

Content truncated to 500 words