Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6149–6157 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6149 r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection Kai Nakamura*¶, Sharon Levy*§, William Yang Wang§ ¶Laguna Blanca School §University of California, Santa Barbara kai.nakamura42@gmail.com, {sharonlevy, william}@cs.ucsb.edu Abstract Fake news has altered society in negative ways in politics and culture. It has adversely affected both online social network systems as well as offline communities and conversations. Using automatic machine learning classification models is an efficient way to combat the widespread dissemination of fake news. However, a lack of effective, comprehensive datasets has been a problem for fake news research and detection model development. Prior fake news datasets do not provide multimodal text and image data, metadata, comment data, and fine-grained fake news categorization at the scale and breadth of our dataset. We present Fakeddit, a novel multimodal dataset consisting of over 1 million samples from multiple categories of fake news. After being processed through several stages of review, the samples are labeled according to 2-way, 3-way, and 6-way classification categories through distant supervision. We construct hybrid text+image models and perform extensive experiments for multiple variations of classification, demonstrating the importance of the novel aspect of multimodality and fine-grained classification unique to Fakeddit. Keywords: fake news, machine learning, multimodal 1. Introduction Within our progressively digitized society, the spread of fake news and disinformation has enlarged in journal- ism, news reporting, social media, and other forms of online information consumption. False information from these sources, in turn, has caused many problems such as spurring irrational fears during medical outbreaks like Ebola1. The dissemination and consequences of fake news are exacerbating due to the rise of popular social media ap- plications and other online sources with inadequate fact- checking or third-party filtering, enabling any individual to broadcast fake news easily and at a large scale (Allcott and Gentzkow, 2017). Though steps have been taken to detect and eliminate fake news, it still poses a dire threat to society (Dreyfuss and Lapowsky, 2019). According to a Pew Re- search Center report2, 50% of Americans view fake news as a critical problem, placing it above violent crime. In ad- dition, the report found that 68% of Americans view fake news as having a significant impact on their confidence of the government and 54% viewed it as having a large impact in their trust in one another. As such, research in the area of fake news detection is of high importance for society. To build a fake news detection model, one must obtain siz- able and diverse training data. Within this area of research, there are several existing published datasets. However, they have many constraints: limited size, modality, and granu- larity. Most conventional fake news research and datasets such as LIAR (Wang, 2017) and Some-Like-It-Hoax (Tac- chini et al., 2017) solely focus on text data. However, on- line information today is also consumed through multime- * …

Content truncated to 500 words