IUCL at SemEval-2016 Task 6: An Ensemble Model for Stance Detection in Twitter

Proceedings of SemEval-2016, pages 394–400, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics IUCL at SemEval-2016 Task 6: An Ensemble Model for Stance Detection in Twitter Can Liu, Wen Li, Bradford Demarest, Yue Chen, Sara Couture, Daniel Dakota, Nikita Haduong, Noah Kaufman, Andrew Lamont, Manan Pancholi, Kenneth Steimel, Sandra K¨ubler Indiana University {liucan,wl9,bdemares,yc59,scouture,ddakota,nhaduong,nokaufma, alamont,mpanchol,ksteimel,skuebler}@indiana.edu Abstract We present the IUCL system, based on super- vised learning, for the shared task on stance detection. Our official submission, the ran- dom forest model, reaches a score of 63.60, and is ranked 6th out of 19 teams. We also use gradient boosting decision trees and SVM and merge all classifiers into an ensemble method. Our analysis shows that random forest is good at retrieving minority classes and gradient boosting majority classes. The strengths of different classifiers wrt. precision and recall complement each other in the ensemble. 1 Introduction Stance detection is a difficult task since it often re- quires reasoning in order to determine whether an utterance is in favor of or against a specific issue. In the shared task (see Mohammad et al. (2016) for de- tails about the shared task), we interpret it as a vari- ant of sentiment analysis and adopt an approach that combines shallow lexical features with an ensemble of different supervised machine learning classifiers. Previous work has shown that using “arguing” fea- tures based on an arguing lexicon along with modal verbs and targets identified via syntactic rules (So- masundaran and Wiebe, 2010); finding polarized re- lations between aspects and topics (Somasundaran and Wiebe, 2009); adding semantic frames (Hasan and Ng, 2013) and contextual features (Anand et al., 2011) generally improve results. Since some of these features do not generalize across targets (Anand et al., 2011), and since we have an additional challenge in processing Twitter data, we rely on uni- gram features and word vectors. This means that our approach is incapable of handling sarcasm or humor. Instead, it provides a robust basis on which we can later add more informative features. Our approach consists of classifiers with a bag of words (unigrams) or with word vectors as features. We use three separate classifiers (SVMs, random forest, gradient boosting decision trees) and an en- semble classifier (TiMBL). Our official submission is the random forest classifier with word unigrams. 2 Methods We use the data sets provided by the SemEval-2016 shared task 6 (Mohammad et al., 2016). 2.1 Preprocessing Preprocessing mostly consists of tokenization. Dur- ing tokenization, we normalize capitalization, and all punctuation signs are separated except for @ and #, as these symbols indicate hashtags and handles. We extract frequency counts of each token in the en- tire corpus and in each stance (Favor, Against, None) per target for use in the feature selection process. We experimented with TWEEBOPARSER (Kong et al., 2014), a dependency parser specifically de- signed for Twitter data, to extract dependency re- lations among words. We extract POS tags, multi- word expressions, and dependency triples from the parses. However, due …

Content truncated to 500 words