Unimodal Intermediate Training for Multimodal Meme Sentiment Classification

Proceedings of Recent Advances in Natural Language Processing, pages 494–506 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_055 494 Unimodal Intermediate Training for Multimodal Meme Sentiment Classification Muzhaffar Hazman1, Susan McKeever2, and Josephine Griffith1 1University of Galway, Ireland 2Technological University Dublin, Ireland {m.hazman1,josephine.griffith}@universityofgalway.ie susan.mckeever@TUDublin.ie Abstract Internet Memes remain a challenging form of user-generated content for automated senti- ment classification. The availability of labelled memes is a barrier to developing sentiment clas- sifiers of multimodal memes. To address the shortage of labelled memes, we propose to sup- plement the training of a multimodal meme classifier with unimodal (image-only and text- only) data. In this work, we present a novel variant of supervised intermediate training that uses relatively abundant sentiment-labelled uni- modal data. Our results show a statistically significant performance improvement from the incorporation of unimodal text data. Further- more, we show that the training set of labelled memes can be reduced by 40% without reduc- ing the performance of the downstream model. 1 Introduction As Internet Memes (or just “memes”) become in- creasingly popular and commonplace across digital communities worldwide, research interest to ex- tend natural language classification tasks, such as sentiment classification, hate speech detection, and sarcasm detection, to these multimodal units of expression has increased. However, state-of-the- art multimodal meme sentiment classifiers signifi- cantly underperform contemporary text sentiment classifiers and image sentiment classifiers. Without accurate and reliable methods to identify the senti- ment of multimodal memes, social media sentiment analysis methods must either ignore or inaccurately infer opinions expressed via memes. As memes continue to be a mainstay in online discourse, our ability to infer the meaning they convey becomes increasingly pertinent (Sharma et al., 2020; Mishra et al., 2023). Achieving similar levels of sentiment classifica- tion performance on memes as on unimodal con- tent remains a challenge. In addition to its multi- modal nature, multimodal meme classifiers must discern sentiment from culturally specific inputs that comprise brief texts, cultural references, and vi- sual symbolism (Nissenbaum and Shifman, 2017). Although various approaches have been used to extract information from each modality (text and image) recent works have highlighted that meme classifiers must also recognise the various forms of interactions between these two modalities (Zhu, 2020; Shang et al., 2021; Hazman et al., 2023). Current approaches to training meme classifiers are dependent on datasets of labelled memes (Kiela et al., 2020; Sharma et al., 2020; Suryawanshi et al., 2020; Patwa et al., 2022; Mishra et al., 2023) containing sufficient samples to train classifiers to extract relevant features from each modality and relevant cross-modal interactions. Relative to the complexity of the task, the current availability of labelled memes still poses a problem, as many cur- rent works call for more data (Zhu, 2020; Kiela et al., 2020; Sharma et al., 2022). Worse still, memes are hard to label. The complexity and culture dependence of memes (Gal et al., 2016) cause the Subjective Perception Problem (Sharma et al., 2020), where varying fa- miliarity and emotional reaction to the contents of a meme from each annotator …

Content truncated to 500 words