Caption Enriched Samples for Improving Hateful Memes Detection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9350–9358 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9350 Caption Enriched Samples for Improving Hateful Memes Detection Efrat Blaier∗ Tel-Aviv University Itzik Malkiel∗ Tel-Aviv University Lior Wolf Tel-Aviv University Abstract The recently introduced hateful meme chal- lenge demonstrates the difficulty of determin- ing whether a meme is hateful or not. Specif- ically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Mo- tivated by the need to model the contrast be- tween the image content and the overlayed text, we suggest applying an off-the-shelf im- age captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the uni- modal case, continuing the pre-training of lan- guage models on augmented and original cap- tion pairs, is highly beneficial to the classifica- tion accuracy. Our code is publicly available 1. 1 Introduction Multimodal transformers, including Visual- BERT (Li et al., 2019), VilBERT (Lu et al., 2019), and UNITER (Chen et al., 2020) are currently the state of the art methods in tasks that involve both text and images, such as visual question answering (Antol et al., 2015) and image caption- ing (Chen et al., 2015). One particular example is that of the “hateful memes” challenge (Kiela et al., 2020), in which the task is to classify whether a given meme is hateful or not. By construction (and similar to real-world memes), the classifier has to apply complex reasoning that involves a multimodal text-image analysis from both image and text. The prevalence of sarcasm adds another layer of difficulty to both machines and humans. Kiela et al. (2020) results indicate that the human accuracy is only about 85%. Recent ∗ Denotes equal contribution. 1https://github.com/efrat-safanov/caption-enriched- samples-research Figure 1: An example of memes in which the classific- tion depends on both the text and the image. Specif- ically, in this Benign Confounder example, the same text appears and the classification changes depending on the image. state-of-the-art models perform considerably less accurately, achieving up to 64.73% accuracy. In this work, we adopt an off-the-shelf caption generator, and show that combining such captions as a third input (in addition to the image and the overlayed text) can effectively improve the perfor- mance of both multimodal and unimodal models. Two techniques are used: continued pre-training of the baseline model with the additional augmented captions, and using the added information as part of the fine-tuned model used for the hateful memes task. 2 Related Work Transformer-based Language Models (LMs) have shown significant performance gains in various nat- ural language understanding tasks, such as text- similarity and language inference (Devlin et al., 2019; Liu et al., 2019). These models are usu- ally pre-trained on the Masked Language Model (MLM) objective followed by a task-specific fine- tuning process (Wang et al., 2018). Recently, vision and language tasks have gained a lot …

Content truncated to 500 words