Mama Edha at SemEval-2017 Task 8: Stance Classification with CNN and Rules

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 481–485, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics Mama Edha at SemEval-2017 Task 8: Stance Classification with CNN and Rules Marianela Garc´ıa Lozano*,†, Hanna Lilja*, Edward Tj¨ornhammar†, and Maja Karasalo* *FOI Swedish Defence Research Agency *{garcia, hanna.lilja, maja.karasalo}@foi.se †KTH Royal Institute of Technology †{mgl, edwardt}@kth.se Abstract For the competition SemEval-2017 we in- vestigated the possibility of performing stance classification (support, deny, query or comment) for messages in Twitter conversation threads related to rumours. Stance classification is interesting since it can provide a basis for rumour veracity as- sessment. Our ensemble classification ap- proach of combining convolutional neural networks with both automatic rule min- ing and manually written rules achieved a final accuracy of 74.9% on the compe- tition’s test data set for Task 8A. To im- prove classification we also experimented with data relabeling and using the gram- matical structure of the tweet contents for classification. 1 Introduction The task of determining the veracity of a rumour is sometimes a difficult one, even with the reasoning power of a human being. This paper presents an approach to an automatic analysis of discussion el- ements with respect to rumours. Discussion struc- ture and analysis can well play a part in a broader effort to assess rumour veracity, and the expecta- tion is that the results presented here is one step of the way towards that end goal. The research presented in this paper is a sub- mission to SemEval-2017, Task 8 (RumourEval: Determining rumour veracity and support for ru- mours), Subtask A (SDQC) (Derczynski et al., 2017). The objective of this subtask is to classify the relation between a tweet and the rumour it is related to in terms of support, deny, query or com- ment. Our approach to this classification task is build- ing three different classifiers and combining the predictions in an ensemble method. The general idea is that different types of classifiers may learn different concepts and hence complement each other, resulting in a better prediction capability for the joint classifier. Furthermore we tested the ac- curacy in applying our ensemble approach to both originally labeled and relabeled data. The remainder of this paper is organized as fol- lows: Section 2 describes the data given in the task and our observations on irregularities in the data labeling. Section 3 contains a description of the process and used methods employed for the ex- periments. Sections 4 and 5 describe the results and discusses the findings. Finally, we conclude the work in Section 6. 2 Data Inconsistencies The data used in the SemEval-2017 Task 8A is a subset of the PHEME data set of social media ru- mours (Zubiaga et al., 2016). When studying the dataset, it was discovered that for some tweets, the annotation is somewhat inconsistent, e.g., tweets with very similar con- tents have sometimes received different annota- tions. For example, despite being nearly identical replies to the same source tweet, the …

Content truncated to 500 words