Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 792–805 November 20–23, 2022. ©2022 Association for Computational Linguistics 792 Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis Shivam Sharma1,4, Mohd Khizir Siddiqui3, Md. Shad Akhtar1 and Tanmoy Chakraborty2 1Indraprastha Institute of Information Technology Delhi, India 2Indian Institute of Technology Delhi, India 3Birla Institute of Technology and Science, Goa, India 4Wipro AI Labs, India {shivams, shad.akhtar}@iiitd.ac.in, mdkhizirsiddiqui@gmail.com, tanchak@ee.iitd.ac.in Abstract Existing self-supervised learning strategies are constrained to either a limited set of objec- tives or generic downstream tasks that predom- inantly target uni-modal applications. This has isolated progress for imperative multi- modal applications that are diverse in terms of complexity and domain-affinity, such as meme analysis. Here, we introduce two self-supervised pre-training methods, namely Ext-PIE-Net and MM-SimCLR that (i) em- ploy off-the-shelf multi-modal hate-speech data during pre-training and (ii) perform self- supervised learning by incorporating multiple specialized pretext tasks, effectively catering to the required complex multi-modal represen- tation learning for meme analysis. We experiment with different self-supervision strategies, including potential variants that could help learn rich cross-modality repre- sentations and evaluate using popular linear probing on the Hateful Memes task. The proposed solutions strongly compete with the fully supervised baseline via label-efficient training while distinctly outperforming them on all three tasks of the Memotion challenge with 0.18%, 23.64%, and 0.93% performance gain, respectively. Further, we demonstrate the generalizability of the proposed solutions by reporting competitive performance on the HarMeme task. Finally, we empirically es- tablish the quality of the learned representa- tions by analyzing task-specific learning, us- ing fewer labeled training samples, and argu- ing that the complexity of the self-supervision strategy and downstream task at hand are cor- related. Our efforts highlight the requirement of better multi-modal self-supervision meth- ods involving specialized pretext tasks for ef- ficient fine-tuning and generalizable perfor- mance. 1 Introduction The overwhelming scale of digital mutation con- stantly transpiring over the web is “creating the illusion of reality, addressing the viewer, and rep- resenting a convoluted space" (Manovich, 2001). Almost every social activity affects or is affected by an online entity, sometimes even disturbing social harmony, influenced by a prominent surge of multi- modal harmful, abusive and hateful online content. Therefore, it is imperative to explore solutions to- wards automatic mediation of online activities that pre-dominantly involve multi-modality. Recently, there has been a defining resurgence of advance- ments in multi-modal AI, albeit slowly. Existing self-supervision strategies for visual- linguistic applications involve different pretext tasks like Masked Language Modeling (MLM) (Devlin et al., 2019), Masked Region Modeling (MRM) (Chen et al., 2020b), Word-Region Align- ment (WRA) (Gupta et al., 2017), and Image- Text Matching (ITM) (Li et al., 2019a; Radford et al., 2021), which inherently presume visual- linguistic grounding (Karpathy and Fei-Fei, 2017). As a consequence, the large-scale datasets like MS COCO (Lin et al., 2014), Conceptual Captions (CC) (Sharma et al., 2018), Wikipedia-based Image Text (WIT) (Srinivasan …

Content truncated to 500 words