SemEval-2016 Task 6: Detecting Stance in Tweets

Proceedings of SemEval-2016, pages 31–41, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 6: Detecting Stance in Tweets Saif M. Mohammad National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Svetlana Kiritchenko National Research Council Canada svetlana.kiritchenko@nrc-cnrc.gc.ca Parinaz Sobhani University of Ottawa psobh090@uottawa.ca Xiaodan Zhu National Research Council Canada xiaodan.zhu@nrc-cnrc.gc.ca Colin Cherry National Research Council Canada colin.cherry@nrc-cnrc.gc.ca Abstract Here for the first time we present a shared task on detecting stance from tweets: given a tweet and a target entity (person, organiza- tion, etc.), automatic natural language systems must determine whether the tweeter is in favor of the given target, against the given target, or whether neither inference is likely. The target of interest may or may not be referred to in the tweet, and it may or may not be the tar- get of opinion. Two tasks are proposed. Task A is a traditional supervised classification task where 70% of the annotated data for a target is used as training and the rest for testing. For Task B, we use as test data all of the instances for a new target (not used in task A) and no training data is provided. Our shared task re- ceived submissions from 19 teams for Task A and from 9 teams for Task B. The highest clas- sification F-score obtained was 67.82 for Task A and 56.28 for Task B. However, systems found it markedly more difficult to infer stance towards the target of interest from tweets that express opinion towards another entity. 1 Introduction Stance detection is the task of automatically deter- mining from text whether the author of the text is in favor of, against, or neutral towards a proposition or target. The target may be a person, an organization, a government policy, a movement, a product, etc. For example, one can infer from Barack Obama’s speeches that he is in favor of stricter gun laws in the US. Similarly, people often express stance to- wards various target entities through posts on online forums, blogs, Twitter, Youtube, Instagram, etc. Au- tomatically detecting stance has widespread appli- cations in information retrieval, text summarization, and textual entailment. The task we explore is formulated as follows: given a tweet text and a target entity (person, or- ganization, movement, policy, etc.), automatic nat- ural language systems must determine whether the tweeter is in favor of the given target, against the given target, or whether neither inference is likely. For example, consider the target–tweet pair: Target: legalization of abortion Tweet: The pregnant are more than walking(1) incubators, and have rights! We can deduce from the tweet that the tweeter is likely in favor of the target.1 We annotated 4870 English tweets for stance to- wards six commonly known targets in the United States. The data corresponding to five of the targets (‘Atheism’, ‘Climate Change is a Real Concern’, ‘Feminist Movement’, ‘Hillary Clinton’, and ‘Le- galization of Abortion’) was used in a standard su- pervised stance detection task – Task A. About 70% of the …

Content truncated to 500 words