Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

Findings of the Association for Computational Linguistics: EACL 2023, pages 399–414 May 2-6, 2023 ©2023 Association for Computational Linguistics Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking Mubashara Akhtar, Oana Cocarascu and Elena Simperl Department of Informatics, King’s College London {mubashara.akhtar,oana.cocarascu,elena.simperl}@kcl.ac.uk Abstract Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video. While there is increasing interest in using images for AFC, previous works mostly focus on detect- ing manipulated or fake images. We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence. ChartBERT lever- ages textual, structural and visual information of charts to determine the veracity of textual claims. For evaluation, we create ChartFC, a new dataset of 15, 886 charts. We system- atically evaluate 75 different vision-language (VL) baselines and show that ChartBERT out- performs VL models, achieving 63.8% accu- racy. Our results suggest that the task is com- plex yet feasible, with many challenges ahead. 1 Introduction Charts are often used to present data in news ar- ticles, reports, scientific publications, and across social media posts (Lo et al., 2022; Zhang et al., 2021). For example, in recent years, charts have been widely used to guide policymakers in decid- ing health policies and to communicate COVID information with the general public; a popular ex- ample is the coronavirus dashboard by Johns Hop- kins University,1 which was integrated in several websites (Perkel, 2020). Misinformation can spread through charts in var- ious ways. Previous works in data visualization have discussed how misleading chart design can cause misinformation (Lo et al., 2022). However, a more subtle form of misinformation occurs during chart interpretation (e.g. through invalid compar- isons, framing correlation as causation, or spread- ing of misleading claims). To identify these mis- information types not only the stand-alone chart but the chart together with its message need to be 1https://coronavirus.jhu.edu/map.html Claim: Both Thane Baker and Nate Cartmell were ranked last. Evidence: Label: Supports Figure 1: An example from the ChartFC dataset where the claim is supported by the evidence chart. considered jointly (Lo et al., 2022). In this work, we focus on verifying whether charts support or refute claims about them. There has been substantial progress in automated fact-checking (AFC) in recent years, with a fo- cus on verifying claims against text (Wang, 2017; Thorne et al., 2018; Schuster et al., 2021; Thorne et al., 2021; Diggelmann et al., 2020), table (Aly et al., 2021; Diggelmann et al., 2020; Chen et al., 2020a; Akhtar et al., 2022), and image (Yao et al., 2022; Zlatkova et al., 2019; Qu et al., 2022) evi- dence. Previous work has widely ignored claim verification against chart images. There are sev- eral challenges related to chart fact-checking as opposed to other evidence modalities: the struc- tural information, text in charts, and location of text need to be considered jointly for chart under- standing. Text plays a key role and is used, for example, as bar …

Content truncated to 500 words