CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3362 - 3376 July 10-15, 2022 ©2022 Association for Computational Linguistics CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking Xuming Hu1∗, Zhijiang Guo2∗, Guanyu Wu1, Aiwei Liu1, Lijie Wen1†, Philip S. Yu1,3 1Tsinghua University 2University of Cambridge 3University of Illinois at Chicago 1{hxm19,wugy18,liuaw20}@mails.tsinghua.edu.cn 2zg283@cam.ac.uk 1wenlj@tsinghua.edu.cn 3psyu@uic.edu Abstract The explosion of misinformation spreading in the media ecosystem urges for automated fact- checking. While misinformation spans both ge- ographic and linguistic boundaries, most work in the field has focused on English. Datasets and tools available in other languages, such as Chinese, are limited. In order to bridge this gap, we construct CHEF, the first CHinese Evidence-based Fact-checking dataset of 10K real-world claims. The dataset covers multiple domains, ranging from politics to public health, and provides annotated evidence retrieved from the Internet. Further, we develop established baselines and a novel approach that is able to model the evidence retrieval as a latent variable, allowing jointly training with the veracity pre- diction model in an end-to-end fashion. Exten- sive experiments show that CHEF will provide a challenging testbed for the development of fact-checking systems designed to retrieve and reason over non-English claims. Source code and data are available1. 1 Introduction Misinformation is being spread online at increas- ing rates, posing a challenge to media platforms from newswire to social media. In order to combat the proliferation of misinformation, fact-checking is an essential task that assesses the veracity of a given claim based on evidence (Vlachos and Riedel, 2014). Fact-checking is commonly con- ducted by journalists. However, fact-checking is a time-consuming task, which can take journalists several hours or days (Adair et al., 2017). Thus, there is a need for automating the process. Although misinformation spans both geographic and linguistic boundaries, most existing works fo- cused on English (Wang, 2017; Thorne et al., 2018; Augenstein et al., 2019; Hanselowski et al., 2019; 1https://github.com/THU-BPM/CHEF ∗Equally Contributed. †Corresponding Author. Claim: 2019年, 共有12.08万人参加成都中考，但招 生计划只有4.3万。In 2019, a total of 120,800 students participated in the high school entrance examination in Chengdu, but schools only enrolled 43,000 students. Document: 今年共有12.08万人参加中考，这个是 成都全市, 包括了20个区，高新区和天府新区的总 参考人数。 月前，教育局公布了2019年的普高招 生计划。招生计划数进一步增加，上普高的机会更 大了... 中心城区（13个区）招生计划为43015人。 This year, 120,800 people participated in the high school entrance examination. This number is for the entire city of Chengdu, including 20 districts, high-tech zone and Tianfu new district. A month ago, the Education Bureau announced the 2019 high school enrollment plan. The number of enrollment will be increased, indicating that there is a greater chance of going to high school... The plan of the central area (including 13 districts) is 43,015. Verdict: Refuted; Domain: Society Challenges: Evidence Collection; Numerical Reasoning Table 1: An example from CHEF (Chinese is translated into English). The claim is refuted by the evidence, which are sentences retrieved (highlighted) from the document. For brevity, only the relevant snippet of the document is shown. Chen et al., 2020). There only exists a handful of non-English datasets for verifying real-world claims. …

Content truncated to 500 words