Memebusters at SemEval-2020 Task 8: Feature Fusion Model for Sentiment Analysis on Memes Using Transfer Learning

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1163–1171 Barcelona, Spain (Online), December 12, 2020. 1163 Memebusters at SemEval-2020 Task 8: Feature Fusion Model for Sentiment Analysis on Memes using Transfer Learning Mayukh Sharma, Ilanthenral Kandasamy, W.B. Vasantha School of Computer Science and Engineering Vellore Institute of Technology Vellore, Tamil Nadu, India mayukh.sharma2016@vitstudent.ac.in, ilanthenral.k@vit.ac.in, vasantha.wb@vit.ac.in Abstract In this paper, we describe our deep learning system used for SemEval 2020 Task 8: Memotion analysis. We participated in all the subtasks i.e Subtask A: Sentiment classification, Subtask B: Humor classification, and Subtask C: Scales of semantic classes. Similar multimodal architecture was used for each subtask. The proposed architecture makes use of transfer learning for images and text feature extraction. The extracted features are then fused together using stacked bi- directional Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) model with attention mechanism for final predictions. We also propose a single model for predicting semantic classes (Subtask B) as well as their scales (Subtask C) by branching the final output of the post LSTM dense layers. Our model was ranked 5 in Subtask B and ranked 8 in Subtask C and performed nicely in Subtask A on the leader board. Our system makes use of transfer learning for feature extraction and fusion of image and text features for predictions. 1 Introduction A meme is an approach, concept, idea or style that spreads through social media within a society often with the goal of expressing a trend, topic, or significance represented by the meme (Peirson and Tolunay, 2018). Recent developments in social media analytics have shown a widespread increase in the usage of memes in expressing sentiment. Social media networks like Facebook, Instagram and Twitter usually get flooded with memes upon occurrence of any popular event. Memes are present in almost every form of media and they are constantly evolving considering the events happening around the world. The spread of memes on social media can be considered analogous to human genetics. Slightly altered versions of the same memes spread in social media in a similar way as copies of human genes pass through human lineage (French, 2017). They work as a medium for sharing humour on the pretext of cultural themes. They represent the sentiment of a community or culture with respect to the event involved. This is beneficial in gauging the sentiment of people. Memes concerning to political events may represent the political outlook of a community. Simultaneously, they can also be engineered to further political ideals, amplify echo chambers and alienate minorities. They are altogether a different means of communication and have become fundamental part of the current generation. Memes are uniquely multimodal, they contain images in conjugation with textual comments and conversations to add emphasis and provide additional meaning. Rarely, memes contain only the text or visual component. Thus, they pose strong challenge in identifying their semantics which may involve use of text as well as image features. Significant research has been carried out on sentiment classification …

Content truncated to 500 words