YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 748 - 755 July 14-15, 2022 ©2022 Association for Computational Linguistics YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT Chao Han, Jin Wang and Xuejie Zhang School of Information Science and Engineering Yunnan University Kunming, China hc_super@mail.ynu.edu.cn, {wangjin, xjzhang}@ynu.edu.cn Abstract This paper describes our system used in the SemEval-2022 Task5 Multimedia Automatic Misogyny Identification (MAMI). This task is to use the provided text-image pairs to clas- sify emotions. In this paper, We propose a multi-label emotion classification model based on pre-trained LXMERT. We use Faster- RCNN to extract visual representation and utilize LXMERT’s cross-attention for multi- modal alignment. Then we use the Bilinear- interaction layer to fuse these features. Our experimental results surpass the F1 score of baseline. For Sub-task A, our F1 score is 0.662 and Sub-task B’s F1 score is 0.633. The code of this study is available on GitHub1. 1 Introduction In social networks, meme is mainly used to ex- press the emotion of netizen. It usually consists of text and images. But at the same time, memes also convey some negative emotions, such as neg- ative comments about women. SemEval-2022 Task5: Multimedia Automatic Misogyny Identi- fication (MAMI) (Fersini et al., 2022) focuses on identifying whether meme conveys negative emo- tions towards women. • Sub-task A: a basic task about misogynous meme identification, where a meme should be categorized either as misogynous or not misogynous; • Sub-task B: an advanced task, where the type of misogyny should be recognized among po- tential overlapping categories such as stereo- type, shaming, objectification, and violence. Since the Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2019) models were pro- posed, researchers have begun to work on image 1https://github.com/HC-super/ SemEval-2022-Task-5 and text multi-modality work in recent years, in addition to using one modality such as only im- age or text. Nowadays, for multimodal models, they can be divided into two categories, single- stream model and dual-stream model. In the single-stream model, language information and vi- sion information are fused at the beginning and directly input into the encoder. Some represen- tative single-stream models include ImageBERT (Qi et al., 2020), Unicoder VL (Li et al., 2020), VL-BERT (Su et al., 2020), VisualBERT (Li et al., 2019), etc. In the dual-stream model, in addition to the LXMERT, we will introduce below, there were ViLBert (Lu et al., 2019) and UNIMO (Li et al., 2021), etc. As for emotion recognition, in previous tasks, there are also emotion classification tasks based on multi-modal graphics and text, such as Zhu et al. (2021) used text-CNN and ALBERT to Iden- tify the persuasion skills of Meme. Peng et al. (2020) used the adversarial learning of sentiment word representations for sentiment analysis. tree-structured regional CNN-LSTM (Wang et al.,A 2020) and dynamic routing in a tree-structured LSTM (Wang et al., 2019) were used for dimen- sional sentiment analysis. In previous SemEval competitions, Tian et al. (2021) extracted hetero- geneous visual representations …

Content truncated to 500 words