SESAM at SemEval-2020 Task 8: Investigating the Relationship between Image and Text in Sentiment Analysis of Memes

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 804–816 Barcelona, Spain (Online), December 12, 2020. 804 SESAM at SemEval-2020 Task 8: Investigating the relationship between image and text in sentiment analysis of memes Lisa Bonheme School of Computing University of Kent Canterbury, CT2 7NF, UK lb732@kent.ac.uk Marek Grzes School of Computing University of Kent Canterbury, CT2 7NF, UK m.grzes@kent.ac.uk Abstract This paper presents our submission to task 8 (memotion analysis) of the SemEval 2020 competition. We explain the algorithms that were used to learn our models along with the process of tuning the algorithms and selecting the best model. Since meme analysis is a challenging task with two distinct modalities, we studied the impact of different multimodal representation strategies. The results of several approaches to dealing with multimodal data are therefore discussed in the paper. We found that alignment-based strategies did not perform well on memes. Our quantitative results also showed that images and text were uncorrelated. Fusion-based strategies did not show significant improvements and using one modality only (text or image) tends to lead to better results when applied with the predictive models that we used in our research. 1 Introduction SemEval 2020 task 8 (Sharma et al., 2020) is a sentiment analysis task targeted at memes1 divided into three sub-tasks of increasing complexity: Sub-task A is predicting the sentiment polarity of a meme, Sub-task B is a multi-label binary classification task which aims to predict whether a meme is humorous, offensive, sarcastic and/or motivational (it can also have neither of these attributes), Sub-task C is a multi-output ordinal classification task which aims to predict the degree of humour, offence, sarcasm and motivation of a meme. The dataset used for this task contains memes images whose text has been extracted by optical character recognition (OCR) and manually corrected when needed. Each meme is annotated on different aspects: sentiment polarity for sub-task A and the degree of humour, sarcasm, offence and motivation for sub-tasks B and C. Memes sentiment analysis is a challenging task as memes are multi-modal, rely heavily on implicit knowledge, and often use humour and sarcasm. While this topic is of growing interest for NLP community, the way image and text interact in memes has barely been explored, leading to sub-optimal representation learning. In an attempt to shed some light on the role of both modalities, we investigate their correlation and their impact on each sub-task prediction. Our code is available at https://github.com/bonheml/SESAM. 2 Related work Sentiment analysis of text is a very active research area which still faces multiple challenges such as irony and humour detection (Hern´andez Farias and Rosso, 2017) and low inter-annotator agreement caused by the high subjectivity of the content (Mohammad, 2017). Research has been extended to multimodal sentiment analysis during the last years (Soleymani et al., 2017), but the focus was mostly on video and text or speech and text. The specific multi-modality of memes in sentiment analysis has only been addressed recently by French (2017), who investigated their correlation …

Content truncated to 500 words