FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7069 - 7086 December 7-11, 2022 ©2022 Association for Computational Linguistics FigMemes: A Dataset for Figurative Language Identification in Politically-Opinionated Memes Chen Liu∗, Gregor Geigle∗†, Robin Krebs, Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP Lab) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt www.ukp.tu-darmstadt.de Abstract Real-world politically-opinionated memes of- ten rely on figurative language to cloak propa- ganda and radical ideas to help them spread. It is not only a scientific challenge to develop machine learning models to recognize them in memes, but also sociologically beneficial to un- derstand hidden meanings at scale and raise awareness. These memes are fast-evolving (in both topics and visuals) and it remains unclear whether current multimodal machine learning models are robust to such distribution shifts. To enable future research into this area, we first present FigMemes, a dataset for figurative lan- guage classification in politically-opinionated memes.1 We evaluate the performance of state-of-the-art unimodal and multimodal mod- els and provide comprehensive benchmark re- sults. The key contributions of this proposed dataset include annotations of six commonly used types of figurative language in politically- opinionated memes, and a wide range of topics and visual styles. We also provide analyses on the ability of multimodal models to generalize across distribution shifts in memes. Our dataset poses unique machine learning challenges and our results show that current models have sig- nificant room for improvement in both perfor- mance and robustness to distribution shifts. The code and dataset (including splits we used for analyses) are available at: https://github. com/UKPLab/emnlp2022-figmemes. Warning: We discuss and show memes that may be offensive to readers for illustrative purposes only. They do not represent the authors’ or the affiliated institution’s views in any way. *Equal Contributions. †Gregor is now affiliated with WüNLP & Computer Vision Lab, CAIDAS, University of Würzburg. 1Disclaimer: The dataset contains racial slurs and other language/images that may be offensive to the readers. This dataset should only be used for academic research or non- commercial purposes. 1 Introduction Memes are transmittable units of culture that evolve fast (Dawkins, 1976), which use images and/or text to convey opinions. Figurative language2 such as metaphors or sarcasm is often used in memes (see Figure 1) to persuade, enhance the impact of ideas, and help these ideas spread (Davison, 2012). This is especially true for internet memes, which rely on figurative language to spread propaganda (Dim- itrov et al., 2021), and support violent, discrimina- tory or radical ideology (Tipler and Ruscher, 2019; Hakoköngäs et al., 2020). For instance, the trans- mission of dehumanized metaphors for women may help perpetuate negative beliefs about women’s roles in modern society (Tipler and Ruscher, 2019), and the usage of rhetorical devices can attract new audiences to radicalized groups that fight against refugees (Hakoköngäs et al., 2020). Figurative language classification has been chal- lenging in machine learning (Wang et al., 2022; Pramanick et al., 2022; Zhang et al., 2021a) as the …

Content truncated to 500 words