Generating Fact Checking Briefs

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7147–7161, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7147 Generating Fact Checking Briefs Angela Fan1,3, Aleksandra Piktus1, Fabio Petroni1, Guillaume Wenzek1, Marzieh Saeidi1, Andreas Vlachos4, Antoine Bordes1, Sebastian Riedel 1,2 1Facebook AI Research 2University College London 3LORIA, 4University of Cambridge {angelafan,piktus,fabiopetroni,guw,marzieh,avlachos,abordes,sriedel}@fb.com Abstract Fact checking at scale is difficult—while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem. How- ever, despite good intentions, contributions from volunteers are often error-prone, and thus in practice restricted to claim detection. We investigate how to increase the accuracy and efficiency of fact checking by providing infor- mation about the claim before performing the check, in the form of natural language briefs. We investigate passage-based briefs, contain- ing a relevant passage from Wikipedia, entity- centric ones consisting of Wikipedia pages of mentioned entities, and Question-Answering Briefs, with questions decomposing the claim, and their answers. To produce QABriefs, we develop QABRIEFER, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. To train its components, we intro- duce QABRIEFDATASET which we collected via crowdsourcing. We show that fact check- ing with briefs — in particular QABriefs — in- creases the accuracy of crowdworkers by 10% while slightly decreasing the time taken. For volunteer (unpaid) fact checkers, QABriefs slightly increase accuracy and reduce the time required by around 20%. 1 Introduction Fact checking is a challenging task. It requires deep knowledge of a claim’s topic and domain, as well as an understanding of the intricacies of misinfor- mation itself. Checking a single claim can take pro- fessional fact checkers 15 minutes to one day (Has- san et al., 2015). Volunteers on the other hand are not considered accurate enough; with access to a search engine, Roitero et al. (2020) report crowd- sourced fact check accuracies of around 58%. This Figure 1: Fact Checking Briefs. Before conducting a fact check, we propose generating briefs to provide information about the claim. We show they make fact checking more accurate and efficient. result corroborates earlier reports1 by fact checking websites which attempted to engage volunteers, but reported success only for claim detection, which is considered a much simpler task (Konstantinovskiy et al., 2018). This is problematic, both from the perspective of using crowdsourced fact checking to combat misinformation and from the perspective of helping individuals fact check themselves. One path for scaling fact checking could be through full automation, taking a claim as input and producing a verdict (Vlachos and Riedel, 2014). Existing work has framed fact checking as classifi- cation, often supported by evidence (Wang, 2017; Thorne et al., 2018; Augenstein et al., 2019). How- ever, due to the limitations of existing automated solutions, practitioners prefer solutions that im- prove efficiency in reaching a verdict, instead of approaches to the complete process (Graves, 2018). In this work, we propose briefs to increase the accuracy …

Content truncated to 500 words