ConStance: Modeling Annotation Contexts to Improve Stance Classification

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1115–1124 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics ConStance: Modeling Annotation Contexts to Improve Stance Classification Kenneth Joseph1 Lisa Friedland1 Will Hobbs1 David Lazer1 Oren Tsur1,2 {k.joseph, l.friedland, w.hobbs}@northeastern.edu d.lazer@neu.edu, orentsur@bgu.ac.il 1Network Science Institute 2Software and Information Systems Engineering Northeastern University Ben Gurion University of the Negev Abstract Manual annotations are a prerequisite for many applications of machine learning. However, weaknesses in the annotation process itself are easy to overlook. In par- ticular, scholars often choose what infor- mation to give to annotators without ex- amining these decisions empirically. For subjective tasks such as sentiment analy- sis, sarcasm, and stance detection, such choices can impact results. Here, for the task of political stance detection on Twit- ter, we show that providing too little con- text can result in noisy and uncertain an- notations, whereas providing too strong a context may cause it to outweigh other signals. To characterize and reduce these biases, we develop ConStance, a gen- eral model for reasoning about annotations across information conditions. Given con- flicting labels produced by multiple an- notators seeing the same instances with different contexts, ConStance simultane- ously estimates gold standard labels and also learns a classifier for new instances. We show that the classifier learned by ConStance outperforms a variety of base- lines at predicting political stance, while the model’s interpretable parameters shed light on the effects of each context. 1 Introduction When annotators are asked for objective judg- ments about a text (e.g., POS tags), the broader context in which the text is situated is often ir- relevant. However, many NLP tasks focus on in- ference of factors beyond words and syntax. For example, the present work addresses the task of detecting political stance on Twitter. We ask an- notators to determine whether a given Twitter user supports Donald Trump or Hillary Clinton. How- ever, inferring something about a user from a sin- gle tweet that she writes may prove difficult. Prior work on stance has relied on annotations collected this way (Mohammad et al., 2016b), but individual tweets do not always contain clear indicators. One solution to this issue is to supply the anno- tator with more information about the user. For ex- ample, for the similar task of classifying a Twitter user’s political affiliation, Cohen and Ruths (2013) display the user’s last 10 tweets. Nguyen et al. (2013), studying gender and age, ask annotators to label users by leveraging all information available in their profile. Thus, researchers have provided a range of contexts (or more broadly, information conditions) to annotators in an attempt to balance annotators’ exposure to the data needed for accu- racy with reasonable costs in terms of time, money and cognitive load. However, while scholars routinely make such decisions about what information to show anno- tators, they rarely examine how such decisions ac- tually impact annotations. The first contribution of this paper (Section 3) is to show that, at least …

Content truncated to 500 words