DialFact: A Benchmark for Fact-Checking in Dialogue

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3785 - 3801 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DialFact: A Benchmark for Fact-Checking in Dialogue Prakhar Gupta†, Chien-Sheng Wu‡, Wenhao Liu‡, Caiming Xiong‡ Language Technologies Institute, Carnegie Mellon University† Salesforce AI Research‡ prakharg@cmu.edu, {wu.jason,wenhao.liu,cxiong}@salesforce.com Abstract Fact-checking is an essential tool to mitigate the spread of misinformation and disinforma- tion. We introduce the task of fact-checking in dialogue, which is a relatively unexplored area. We construct DIALFACT, a testing bench- mark dataset of 22,245 annotated conversa- tional claims, paired with pieces of evidence from Wikipedia. There are three sub-tasks in DIALFACT: 1) Verifiable claim detection task distinguishes whether a response carries verifi- able factual information; 2) Evidence retrieval task retrieves the most relevant Wikipedia snip- pets as evidence; 3) Claim verification task pre- dicts a dialogue response to be supported, re- futed, or not enough information. We found that existing fact-checking models trained on non-dialogue data like FEVER (Thorne et al., 2018) fail to perform well on our task, and thus, we propose a simple yet data-efficient solution to effectively improve fact-checking performance in dialogue. We point out unique challenges in DIALFACT such as handling the colloquialisms, coreferences and retrieval am- biguities in the error analysis to shed light on future research in this direction1. 1 Introduction Misinformation online can have deleterious con- sequences to our society, especially during public health crises like the COVID-19 pandemic. False and outdated information can be spread not only by humans but also by automatic agents as gen- erative models have shown remarkable progress recently (Adiwardana et al., 2020; Xu et al., 2021). These systems are not perfect, as they can either generate hallucinated and imperfect information, or they can be abused to automatically generate false claims and spread misinformation at a mas- sive scale. Fact verification tools are thus necessary in the current information age to tackle the spread of misinformation propagated. 1Data and code are available at https://github. com/salesforce/DialFact Dialogue Context: I have family in Ireland! Have you ever been there? Evidence: Ireland is an island in the North Atlantic. Non-Verifiable Response: I haven’t been but want to! Verifiable Supported Response: I haven’t. It is an island in the north Atlantic right? Verifiable Refuted Response: I haven’t been. Isn’t it somewhere in north Pacific? Verifiable NEI Response: I haven’t been. I heard it’s the most popular tourist location in Europe! Figure 1: Dialogue fact-checking involves predicting if a response should be considered a Verifiable claim, followed by finding relevant evidence, and finally pre- dicting if the it is SUPPORTED, REFUTED or NEI. Fact-checking was introduced in Wang (2017); Thorne et al. (2018) and since then a growing body of research has explored and suggested various tasks and resources to address the challenges in this area. Fact-checking has been explored in medium such as Wikipedia passages, tables, social media and news articles (Guo et al., 2021; Bekoulis et al., 2021). In dialogue domain, related …

Content truncated to 500 words