Generating Literal and Implied Subquestions to Fact-check Complex Claims
31141Authors
nan
Abstract
AbstractVerifying political claims is a challenging task, as politicians can use various tactics to subtly misrepresent the facts for their agenda. Existing automatic fact-checking systems fall short here, and their predictions like “half-true” are not very useful in isolation, since it is unclear which parts of a claim are true and which are not. In this work, we focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present CLAIMDECOMP, a dataset of decompositions for over 1000 claims. Given a claim and its verification paragraph written by fact-checkers, our trained annotators write subquestions covering both explicit propositions of the original claim and its implicit facets, such as asking about additional political context that changes our view of the claim’s veracity. We study whether state-of-the-art models can generate such subquestions, showing that these models generate reasonable questions to ask, but predicting the comprehensive set of subquestions from the original claim without evidence remains challenging. We further show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers, suggesting that they can be useful pieces of a fact-checking pipeline.
Keywords
Full Content
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495 - 3516 December 7-11, 2022 ©2022 Association for Computational Linguistics Generating Literal and Implied Subquestions to Fact-check Complex Claims Jifan Chen Aniruddh Sriram Eunsol Choi Greg Durrett Department of Computer Science The University of Texas at Austin jfchen@cs.utexas.edu Abstract Verifying political claims is a challenging task, as politicians can use various tactics to subtly misrepresent the facts for their agenda. Exist- ing automatic fact-checking systems fall short here, and their predictions like “half-true” are not very useful in isolation, since it is unclear which parts of a claim are true or false. In this work, we focus on decomposing a complex claim into a comprehensive set of yes-no sub- questions whose answers influence the veracity of the claim. We present CLAIMDECOMP, a dataset of decompositions for over 1000 claims. Given a claim and its verification paragraph written by fact-checkers, our trained annota- tors write subquestions covering both explicit propositions of the original claim and its im- plicit facets, such as additional political context that changes our view of the claim’s veracity. We study whether state-of-the-art pre-trained models can learn to generate such subquestions. Our experiments show that these models gener- ate reasonable questions, but predicting implied subquestions based only on the claim (without consulting other evidence) remains challeng- ing. Nevertheless, we show that predicted sub- questions can help identify relevant evidence to fact-check the full claim and derive the ve- racity through their answers, suggesting that claim decomposition can be a useful piece of a fact-checking pipeline.1 1 Introduction Despite a flurry of recent research on automated fact-checking (Wang, 2017; Rashkin et al., 2017; Volkova et al., 2017; Ferreira and Vlachos, 2016; Popat et al., 2017; Tschiatschek et al., 2018), we remain far from building reliable fact-checking sys- tems (Nakov et al., 2021). This challenge motivated us to build more explainable models so the expla- nations can at least help a user interpret the results 1We release our code and dataset: https://jifan-chen. github.io/ClaimDecomp Joe Biden stated on August 31, 2020 in a speech: "When I was vice president, violent crime fell 15% in this country. ... The murder rate now is up 26% across the nation this year under Donald Trump." Claim&Decomposi-on:&focus&of&this&work Claim Q1: Did the crime rate fall by 15% during Joe Biden's presidency? Q2: Did the murder rate in 2020 increase by 26% from 2019? Q3: Is Biden comparing crime rates from the same time interval in his statement? Q4: Is violent crime rate and murder rate directly comparable? Literal Implied } <latexit sha1_base64="/T2 d263pRFT+aJ4AFmd9W0eXiwI=">ACYXicdZFda9swFIZl92 Ot+Vml70RC6EpMHuCu1laXYx2C4y2rSFOBhZlhNRyTLScak x/pO7281u9kempIF27XpA8PC+53CkV0khuIEg+OW4K6tr6x82N r2t7Z3dPX+/dWNUqSkbUSWUvkuIYLnbAQcBLsrNCMyEew2uR /M/dsHpg1X+TVUBZtIMs15xikBK8X+YyeSBGZJVs+auF6wlvX g+1XTeJ1IqCkexhHMGJDu1SGOJE/xlx7+duR13tW7VcyflR6u 4rAXiVSB6VxzY/D5siLmthvB/1gUfgthEto2UNY/9nlCpaS pYDFcSYcRgUMKmJBk4Fa7yoNKwg9J5M2dhiTiQzk3qRUIM7Vk lxprQ9OeCF+nKiJtKYSia2cx6Be3Nxf954xKy80nN86IEltOn RVkpMCg8jxunXDMKorJAqOb2rpjOiCYU7Kd4NoTw9ZPfws1JP /zcP/lx2r64XMaxgQ7QJ9RFITpDF+grGqIRoui3s+rsOLvOH3 fT9d3WU6vrLGc+on/KPfgLUqwkA=</latexit> Judgment:& half-true Aggrega-on evidence+per+ subques2on retrieval (Proof-of-concept:+from+jus2fica2on+doc) Figure 1: An example claim decomposition: the top two subquestions follow explicitly from the claim and the bottom two represent implicit reasoning needed to verify the claim. We can use the decomposed questions to retrieve relevant evidence (Section 6), and aggregate the decisions of the sub-questions to derive the final veracity …
Content truncated to 500 words
Labeled Text
AbstractVerifying political claims is a challenging task, as politicians can use various tactics to subtly misrepresent the facts for their agenda. Existing automatic fact-checking systems fall short here, and their predictions like “half-true” are not very useful in isolation, since it is unclear which parts of a claim are true and which are not. In this work, we focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present CLAIMDECOMP, a dataset of decompositions for over 1000 claims. Given a claim and its verification paragraph written by fact-checkers, our trained annotators write subquestions covering both explicit propositions of the original claim and its implicit facets, such as asking about additional political context that changes our view of the claim’s veracity. We study whether state-of-the-art models can generate such subquestions, showing that these models generate reasonable questions to ask, but predicting the comprehensive set of subquestions from the original claim without evidence remains challenging. We further show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers, suggesting that they can be useful pieces of a fact-checking pipeline.
Publication Details
- Paper ID:
- 31141
- Year:
- 2022
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan