A Survey on Multimodal Disinformation Detection

Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643 October 12–17, 2022. 6625 A Survey on Multimodal Disinformation Detection Firoj Alam1, Stefano Cresci2, Tanmoy Chakraborty3∗ , Fabrizio Silvestri4, Dimitar Dimitrov5, Giovanni Da San Martino6, Shaden Shaar1, Hamed Firooz7, Preslav Nakov8 1Qatar Computing Research Institute, HBKU, 2IIT-CNR, Pisa, 3IIT Delhi, 4Sapienza University of Rome, Italy, 5Sofia University, 6University of Padova, Italy, 7Facebook AI, 8Mohamed bin Zayed University of Artificial Intelligence {fialam, sshaar}@hbku.edu.qa, s.cresci@iit.cnr.it, tanchak@ee.iitd.ac.in, fsilvestri@diag.uniroma1.it mitko.bg.ss@gmail.com, dasan@math.unipd.it,mhfirooz@fb.com, preslav.nakov@mbzuai.ac.ae Abstract Recent years have witnessed the proliferation of offensive content online such as fake news, propaganda, misinformation, and disinforma- tion. While initially this was mostly about textual content, over time images and videos gained popularity, as they are much easier to consume, attract more attention, and spread fur- ther than text. As a result, researchers started leveraging different modalities and combina- tions thereof to tackle online multimodal offen- sive content. In this study, we offer a survey on the state-of-the-art on multimodal disinforma- tion detection covering various combinations of modalities: text, images, speech, video, social media network structure, and temporal infor- mation. Moreover, while some studies focused on factuality, others investigated how harmful the content is. While these two components in the definition of disinformation – (i) factuality, and (ii) harmfulness –, are equally important, they are typically studied in isolation. Thus, we argue for the need to tackle disinformation de- tection by taking into account multiple modali- ties as well as both factuality and harmfulness, in the same framework. Finally, we discuss cur- rent challenges and future research directions. 1 Introduction The proliferation of online social media has en- couraged individuals to freely express their opin- ions and emotions. On one hand, the freedom of speech has led to a massive growth of online con- tent which, if systematically mined, can be used for citizen journalism, public awareness, political campaigning, etc. On the other hand, its misuse has given rise to the proliferation of hostility online (Brooke, 2019; Joksimovic et al., 2019), resulting in offensive content in the form of fake news, hate speech (Schmidt and Wiegand, 2017a; Davidson et al., 2017), propaganda (Da San Martino et al., ∗Work done while T. Chakraborty was at IIIT-Delhi, In- dia. 2019), cyberbullying (Van Hee et al., 2015), etc. Indeed, researchers have argued that this situation has set the dawn of the Post-Truth Era, dominated by emotions and “alternative facts” (Lewandowsky et al., 2017; Cooke, 2018; Nakov and Da San Mar- tino, 2020). More recently, with the emergence of the COVID-19 pandemic, a new blending of medical and political false information has given rise to the first global infodemic (Paka et al., 2021; Zarocostas, 2020; Patwa et al., 2021).1 The term “fake news” is commonly used, al- though it is very generic, and misleads people to focus only on veracity. That is why international or- ganizations such as the UN, WHO, EU, and NATO prefer the term disinformation (Ireton and Posetti, 2018), which refers to information that is (i) fake and also (ii) …

Content truncated to 500 words