A Dataset for Detecting Stance in Tweets

A Dataset for Detecting Stance in Tweets Saif M. Mohammad1, Svetlana Kiritchenko1, Parinaz Sobhani2, Xiaodan Zhu1, Colin Cherry1 1National Research Council Canada, 2University of Ottawa saif.mohammad@nrc-cnrc.gc.ca, svetlana.kiritchenko@nrc-cnrc.gc.ca, psobh090@uottawa.ca, xiaodan.zhu@nrc-cnrc.gc.ca, colin.cherry@nrc-cnrc.gc.ca Abstract We can often detect from a person’s utterances whether he/she is in favor of or against a given target entity (a product, topic, another person, etc.). Here for the first time we present a dataset of tweets annotated for whether the tweeter is in favor of or against pre-chosen targets of interest—their stance. The targets of interest may or may not be referred to in the tweets, and they may or may not be the target of opinion in the tweets. The data pertains to six targets of interest commonly known and debated in the United States. Apart from stance, the tweets are also annotated for whether the target of interest is the target of opinion in the tweet. The annotations were performed by crowdsourcing. Several techniques were employed to encourage high-quality annotations (for example, providing clear and simple instructions) and to identify and discard poor annotations (for example, using a small set of check questions annotated by the authors). This Stance Dataset, which was subsequently also annotated for sentiment, can be used to better understand the relationship between stance, sentiment, entity relationships, and textual inference. Keywords: Stance, argumentation, tweets, sentiment, opinion, target of opinion 1. Introduction Stance detection is the task of automatically determining from text whether the author of the text is in favor of, against, or neutral towards a proposition or target. The tar- get may be a person, an organization, a government policy, a movement, a product, etc. For example, one can infer from Barack Obama’s speeches that he is in favor of stricter gun laws in the US. Similarly, people often express stance towards various target entities through posts on online fo- rums, blogs, Twitter, Youtube, Instagram, etc. Automatically detecting stance has widespread applications in information retrieval, text summarization, and textual en- tailment. Over the last decade, there has been active re- search in modeling stance. However, most work focuses on congressional debates (Thomas et al., 2006) or debates in online forums (Somasundaran and Wiebe, 2009; Mu- rakami and Raymond, 2010; Anand et al., 2011; Walker et al., 2012; Hasan and Ng, 2013b; Sridhar et al., 2014). The task we explore is detecting stance from tweets, and it is formulated as follows: given a tweet and a target en- tity (person, organization, movement, policy, etc.), can au- tomatic natural language systems determine whether the tweeter is in favor of the given target, against the given tar- get, or whether neither inference is likely? For example, consider the target–tweet pair: Target: legalization of abortion Tweet: A foetus has rights too! Make your voice heard. Humans can deduce from the tweet that the tweeter is likely against the target (the tweeter’s stance is against the legal- ization of abortion). Our goal is to create labeled training and test data that can be used in the developments …

Content truncated to 500 words