ParsFEVER: a Dataset for Farsi Fact Extraction and Verification

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 99–104 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 99 ParsFEVER: a Dataset for Farsi Fact Extraction and Verification Majid Zarharan†, Mahsa Ghaderan†, Amin Pourdabiri†, Zahra Sayedi† Behrouz Minaei-Bidgoli†, Sauleh Eetemadi†, and Mohammad Taher Pilehvar‡ †Iran University of Science and Technology, Tehran, Iran ‡Tehran Institute for Advanced Studies, Tehran, Iran {majid_zarharan,m_ghaderan,amin_pourdabiri,s_sayedi}@ comp.iust.ac.ir, {b_minaei,sauleh}@iust.ac.ir, mp792@cam.ac.uk Abstract Training and evaluation of automatic fact ex- traction and verification techniques require large amounts of annotated data which might not be available for low-resource languages. This paper presents ParsFEVER: the first pub- licly available Farsi dataset for fact extrac- tion and verification. We adopt the construc- tion procedure of the standard English dataset for the task, i.e., FEVER, and improve it for the case of low-resource languages. Specif- ically, claims are extracted from sentences that are carefully selected to be more infor- mative. The dataset comprises nearly 23K manually-annotated claims. Over 65% of the claims in ParsFEVER are many-hop (re- quire evidence from multiple sources), mak- ing the dataset a challenging benchmark (only 13% of the claims in FEVER are many-hop). Also, despite having a smaller training set (around one-ninth of that in Fever), a model trained on ParsFEVER attains similar down- stream performance, indicating the quality of the dataset. We release the dataset and the annotation guidelines at https://github. com/Zarharan/ParsFEVER. 1 Introduction The spread of false information can lead to severe social and political problems (Wang, 2017). It would be extremely difficult to detect and track false information manually, given that the abun- dance of available technology has made it possible for these to be produced at scale and disseminated rapidly. Therefore, there has been a lot of inter- est in developing natural language technologies for fact-checking (Thorne and Vlachos, 2018). Unfor- tunately, similarly to many other fields of NLP that rely on manually curated datasets, fact-checking has remained restricted to a few high-resource lan- guages for which large-scale annotated datasets are available. In this paper, we present ParsFEVER, the first Farsi fact extraction and verification dataset. The dataset opens room for research in fact-checking and verification on low-resourced languages. Pars- FEVER is constructed based on FEVER (Thorne et al., 2018), the most widely used dataset for fact- checking and fake news detection in English. We collected 22,906 claims by altering sentences ex- tracted from introductory sections of 358 popular articles from Farsi Wikipedia. Annotators manually classified these claims into SUPPORTED, REFUTED, or NOTENOUGHINFO based on the provided ref- erence pages. In addition, the annotators tagged those sentences which they used as evidence for this classification. Therefore, the dataset can be used for both fact-checking (a 3-class classification task) and evidence retrieval (which is a necessary step for the classification). The quality of the dataset was evaluated using three different validation checks: (1) 5-way inter- annotator agreement, (2) agreement against super- annotators1, and (3) manual validation by the au- thors. We also report experimental results for when ParsFEVER was used …

Content truncated to 500 words