Measuring What Counts: The Case of Rumour Stance Classification

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 925–932 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 925 Measuring What Counts: The Case of Rumour Stance Classification Carolina Scarton University of Sheffield, UK c.scarton@sheffield.ac.uk Diego Furtado Silva Federal University of S˜ao Carlos, Brazil diegofs@ufscar.br Kalina Bontcheva University of Sheffield, UK k.bontcheva.sheffield.ac.uk Abstract Stance classification can be a powerful tool for understanding whether and which users be- lieve in online rumours. The task aims to auto- matically predict the stance of replies towards a given rumour, namely support, deny, ques- tion, or comment. Numerous methods have been proposed and their performance com- pared in the RumourEval shared tasks in 2017 and 2019. Results demonstrated that this is a challenging problem since naturally occur- ring rumour stance data is highly imbalanced. This paper specifically questions the evalua- tion metrics used in these shared tasks. We re- evaluate the systems submitted to the two Ru- mourEval tasks and show that the two widely adopted metrics – accuracy and macro-F1 – are not robust for the four-class imbalanced task of rumour stance classification, as they wrongly favour systems with highly skewed accuracy towards the majority class. To over- come this problem, we propose new evaluation metrics for rumour stance detection. These are not only robust to imbalanced data but also score higher systems that are capable of recognising the two most informative minority classes (support and deny). 1 Introduction The automatic analysis of online rumours has emerged as an important and challenging Natu- ral Language Processing (NLP) task. Rumours in social media can be defined as claims that cannot be verified as true or false at the time of posting (Zubiaga et al., 2018). Prior research (Mendoza et al., 2010; Kumar and Carley, 2019) has shown that the stances of user replies are often a useful predictor of a rumour’s likely veracity, specially in the case of false rumours that tend to receive a higher number of replies denying them (Zubiaga et al., 2016). However, their automatic classifi- cation is far from trivial as demonstrated by the results of two shared tasks – RumourEval 2017 and 2019 (Derczynski et al., 2017; Gorrell et al., 2019). More specifically, sub-task A models ru- mour stance classification (RSC) as a four-class problem, where replies can: • support/agree with the rumour; • deny the veracity of the rumour; • query/ask for additional evidence; • comment without clear contribution to assess- ing the veracity of the rumour. Figure 1 shows an example of a reply denying a post on Twitter. Figure 1: Example of a deny stance. In RumourEval 2017 the training data contains 297 rumourous threads about eight events. The test set has 28 threads, with 20 threads about the same events as the training data and eight threads about unseen events. In 2019, the 2017 training data is augmented with 40 Reddit threads. The new 2019 test …

Content truncated to 500 words