Estimating predictive uncertainty for rumour verification models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6964–6981 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6964 Estimating Predictive Uncertainty for Rumour Verification Models Elena Kochkina University of Warwick, UK The Alan Turing Institute, UK E.Kochkina@warwick.ac.uk Maria Liakata Queen Mary University of London, UK The Alan Turing Institute, UK University of Warwick, UK mliakata@turing.ac.uk Abstract The inability to correctly resolve rumours cir- culating online can have harmful real-world consequences. We present a method for incor- porating model and data uncertainty estimates into natural language processing models for automatic rumour verification. We show that these estimates can be used to filter out model predictions likely to be erroneous, so that these difficult instances can be prioritised by a hu- man fact-checker. We propose two methods for uncertainty-based instance rejection, super- vised and unsupervised. We also show how uncertainty estimates can be used to interpret model performance as a rumour unfolds. 1 Introduction One of the greatest challenges of the information age is the rise of pervasive misinformation. Social media platforms enable it to spread rapidly, reach- ing wide audiences before manual verification can be performed. Hence there is a strive to create automated tools that assist with rumour resolution. Information about unfolding real-world events such as natural disasters often appears in a piece-wise manner, making verification a time-sensitive prob- lem. Failure to identify misinformation can have a harmful impact, thus it is desirable that an auto- mated system aiding rumour verification does not only make a judgement but that it can also inform a human fact-checker of its uncertainty. Deep learning models are currently the state- of-the-art in many Natural Language Processing (NLP) tasks, including rumour detection (Ma et al., 2018), the task of identifying candidate rumours, and rumour verification (Li et al., 2019; Zhang et al., 2019), where the goal is to resolve the verac- ity of a rumour. Latent features and large param- eter spaces of deep learning models make it hard to interpret a model’s decisions. Increasingly re- searchers are investigating methods for understand- ing model predictions, such as through analysing neural attention (Vaswani et al., 2017) and studying adversarial examples (Yuan et al., 2019). Another way to gain insights into a model’s decisions is via estimating its uncertainty. Understanding what a model does not know can help us determine when we can trust its output and at which stage informa- tion needs to be passed on to a human (Kendall and Gal, 2017). In this paper, rather than purely focusing on the performance of a rumour verification model, we es- timate its predictive uncertainty to gain understand- ing of a model’s decisions and filter out the cases that are ’hard’ for the model. We consider two types of predictive uncertainty: data uncertainty (aleatoric) and model uncertainty (epistemic). The approach we adopt requires minimal changes to a given model and is relatively computationally inex- pensive, thus making it possible to apply to various architectures. We make the following contributions: • …

Content truncated to 500 words