SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 69–76, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours Leon Derczynski♥∗ and Kalina Bontcheva♥ and Maria Liakata♣ and Rob Procter♣ and Geraldine Wong Sak Hoi♦ and Arkaitz Zubiaga♣ ♥: Department of Computer Science, University of Sheffield, S1 4DP, UK ♣: Department of Computer Science, University of Warwick, CV4 7AL, UK ♦: swissinfo.ch, Bern, Switzerland ∗: leon.d@shef.ac.uk Abstract Media is full of false claims. Even Ox- ford Dictionaries named “post-truth” as the word of 2016. This makes it more important than ever to build systems that can identify the veracity of a story, and the nature of the discourse around it. Ru- mourEval is a SemEval shared task that aims to identify and handle rumours and reactions to them, in text. We present an annotation scheme, a large dataset cov- ering multiple topics – each having their own families of claims and replies – and use these to pose two concrete challenges as well as the results achieved by partici- pants on these challenges. 1 Introduction and Motivation Rumours are rife on the web. False claims affect people’s perceptions of events and their behaviour, sometimes in harmful ways. With the increasing reliance on the Web – social media, in particular – as a source of information and news updates by in- dividuals, news professionals, and automated sys- tems, the potential disruptive impact of rumours is further accentuated. The task of analysing and determining veracity of social media content has been of recent interest to the field of natural language processing. After initial work (Qazvinian et al., 2011), increasingly advanced systems and annotation schemas have been developed to support the analysis of rumour and misinformation in text (Kumar and Geethaku- mari, 2014; Zhang et al., 2015; Shao et al., 2016; Zubiaga et al., 2016b). Veracity judgment can be decomposed intuitively in terms of a compar- ison between assertions made in – and entailments from – a candidate text, and external world knowl- edge. Intermediate linguistic cues have also been shown to play a role. Critically, based on recent work the task appears deeply nuanced and very challenging, while having important applications in, for example, journalism and disaster mitigation (Hermida, 2012; Procter et al., 2013a; Veil et al., 2011). We propose a shared task where participants analyse rumours in the form of claims made in user-generated content, and where users respond to one another within conversations attempting to resolve the veracity of the rumour. We define a ru- mour as a “circulating story of questionable verac- ity, which is apparently credible but hard to verify, and produces sufficient scepticism and/or anxiety so as to motivate finding out the actual truth” (Zu- biaga et al., 2015b). While breaking news unfold, gathering opinions and evidence from as many sources as possible as communities react becomes crucial to determine the veracity of rumours and consequently reduce the impact of the …

Content truncated to 500 words