Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14571–14589 July 9-14, 2023 ©2023 Association for Computational Linguistics Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation Kung-Hsiang Huang♠ Kathleen McKeown♣ Preslav Nakov♢ Yejin Choi♡♦ Heng Ji♠ ♠ UIUC ♣ Columbia University ♡ University of Washington ♢ MBZUAI ♦AI2 {khhuang3, hengji}@illinois.edu kathy@columbia.edu preslav.nakov@mbzuai.ac.ae yejinc@allenai.org Abstract Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation. What lim- its the successful transfer between them is the sizable gap between machine-generated fake news and human-authored ones, including the notable differences in terms of style and under- lying intent. With this in mind, we propose a novel framework for generating training ex- amples that are informed by the known styles and strategies of human-authored propaganda. Specifically, we perform self-critical sequence training guided by natural language inference to ensure the validity of the generated arti- cles, while also incorporating propaganda tech- niques, such as appeal to authority and loaded language. In particular, we create a new train- ing dataset, PROPANEWS, with 2,256 exam- ples, which we release for future use. Our ex- perimental results show that fake news detec- tors trained on PROPANEWS are better at de- tecting human-written disinformation by 3.62– 7.69% F1 score on two public datasets.1 1 Introduction The dissemination of false information online can cause chaos, hatred, and trust issues, and can eventually hinder the development of society as a whole (Dewatana and Adillah, 2021; Wasserman and Madrid-Morales, 2019). In particular, human- written disinformation2 is often used to manipulate certain populations and reportedly already had a catastrophic impact on multiple events, such as Brexit (Bastos and Mercea, 2019), the COVID-19 pandemic (van Der Linden et al., 2020), and the 2022 Russian assault on Ukraine. 1The code and data are released on GitHub: https:// github.com/khuangaf/FakingFakeNews 2There are many types and definitions of fake news, but here we focus on text-only disinformation. Yet, we will also use the less accurate term fake news as it is more common. Hence, there is an urgent need for a defense mechanism against human-written disinformation.3 For this, we need a substantial amount of training data to build detectors. A naïve solution is to collect human-written news articles that contain inaccu- rate information by crawling untrustworthy news media. However, news articles published by suspi- cious sources do not necessarily contain false infor- mation, which means that annotators are required to fact-check every claim in each untrustworthy ar- ticle. Moreover, articles containing false claims are often removed shortly after posting. While some work collected human-written fake news from fact- checking websites (Shu et al., 2018; Nguyen et al., 2020), the size of these datasets is limited. The curation process of these websites also requires a lot of manual efforts. Hence, such a solution is neither scalable nor reliable. Thus, an alternative direction complementing the existing efforts would be to generate training data automatically in …

Content truncated to 500 words