SRCB at SemEval-2022 Task 5: Pretraining Based Image to Text Late Sequential Fusion System for Multimodal Misogynous Meme Identification

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 585 - 596 July 14-15, 2022 ©2022 Association for Computational Linguistics SRCB at SemEval-2022 Task 5: Pretraining Based Image to Text Late Sequential Fusion System for Multimodal Misogynous Meme Identification Yujin Wang∗, Jing Zhang�, Bohua Peng∗, Xudong Zhang∗ Xiaoyan Qu, Yimeng Zhuang, Song Liu Samsung Research China-Beijing (SRC-B) {yujin1.wang, jing97.zhang, bohua.peng}@samsung.com {xudong.zhang, xiaoyan11.qu}@samsung.com {ym.zhuang, s0101.liu}@samsung.com Abstract Online misogyny meme detection is an im- age/text multimodal classification task, the complicated relation of image and text chal- lenges the intelligent system’s modality fusion learning capability. In this paper, we investigate the single-stream UNITER and dual-stream CLIP multimodal pretrained models on their ca- pability to handle strong and weakly correlated image/text pairs. The XGBoost classifier with image features extracted by the CLIP model has the highest performance and being robust on domain shift. Based on this, we propose the PBR system, an ensemble system of Pretrain- ing models, Boosting method and Rule-based adjustment, text information is fused into the system using our late sequential fusion scheme. Our system ranks 1st place on both sub-task A and sub-task B of the SemEval-2022 Task 5 Multimedia Automatic Misogyny Identifica- tion, with 0.834/0.731 macro F1 scores for sub- task A/B correspondingly. 1 Introduction Much of the real world’s information comes in mul- timodality, a combination of images, texts, audios and so on. Multimodal understanding aims to uti- lize different modal of information to improve the overall system recognition intelligence or robust- ness (Gadzicki et al., 2020), which plays a key foundation role in cognitive AI and embodied AI. With transfer learning by large deep models and colossal corpus achieving remarkable success in vi- sion and language domain, there is a rising interest in combining both sides’ advances to push the mul- timodality understanding further (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2019; Li et al., 2020; Yu et al., 2020; Huo et al., 2021; Kim et al., 2021; Radford et al., 2021). We will limit the dis- cussion scope of multimodal to vision and language in this paper. There are two kinds of representative ∗Contribution during Intership in Samsung Research China-Beijing. architecture of multimodal learning models, single- stream models and dual-stream models. Single- stream model fuses the image and text data at an early stage, and then feed into the model. Dual- stream models design separated structure as image encoder and text encoder, and a further module is stacked on top of the unimodel encoders for cross- modal learning objectives (Tan and Bansal, 2019; Yu et al., 2020; Radford et al., 2021; Huo et al., 2021). Usually per-unimodal objectives and mul- timodal objectives are designed to ensure that the model learns unimodal and crossmodal knowledge, like masked image prediction, masked token predic- tion, and text-image pairing (Chen et al., 2019; Kim et al., 2021). Two kinds of data distributions are explored for the large-scale pretraining, strongly paired data (Chen et al., 2019; Radford et al., 2021; Li et al., 2020; Kim …

Content truncated to 500 words