TIB-VA at SemEval-2022 Task 5: A Multimodal Architecture for the Detection and Classification of Misogynous Memes

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 756 - 760 July 14-15, 2022 ©2022 Association for Computational Linguistics TIB-VA at SemEval-2022 Task 5: A Multimodal Architecture for the Detection and Classification of Misogynous Memes Sherzod Hakimov1,2, Gullal S. Cheema1, and Ralph Ewerth1,2 1TIB – Leibniz Information Centre for Science and Technology 2Leibniz University Hannover, L3S Research Center Hannover, Germany {sherzod.hakimov, gullal.cheema, ralph.ewerth}@tib.eu Abstract The detection of offensive, hateful content on social media is a challenging problem that af- fects many online users on a daily basis. Hate- ful content is often used to target a group of people based on ethnicity, gender, religion and other factors. The hate or contempt to- ward women has been increasing on social platforms. Misogynous content detection is especially challenging when textual and visual modalities are combined to form a single con- text, e.g., an overlay text embedded on top of an image, also known as meme. In this pa- per, we present a multimodal architecture that combines textual and visual features to detect misogynous memes. The proposed architec- ture is evaluated in the SemEval-2022 Task 5: MAMI - Multimedia Automatic Misogyny Identification challenge under the team name TIB-VA. We obtained the best result in the Task- B where the challenge is to classify whether a given document is misogynous and further identify the following sub-classes: shaming, stereotype, objectification, and violence. 1 Introduction Detection of hate speech has become a fundamen- tal problem for many social media platforms such as Twitter, Facebook, and Instagram. There have been many efforts by the research community and companies to identify the applicability of advanced solutions. In general, hate speech is defined as a hateful language targeted at a group or individuals based on specific characteristics such as religion, ethnicity, origin, sexual orientation, gender, physi- cal appearance, disability or disease. The hatred or contempt expressed towards women has been drastically increasing, as reported by Plan Interna- tional (2020) and Vogels (2021). Detection of such misogynous content requires large-scale automatic solutions (Gasparini et al., 2018; Suryawanshi et al., 2020; Menini et al., 2020) and comprehensive an- notation processes (Zeinert et al., 2021). The detection of hateful content has been mainly studied from the textual perspective based on the Computational Linguistics and Natural Language Processing (NLP) fields. However, hateful content on social media can be found in other forms, such as videos, a combination of text and images, or emoticons. Misogynous content detection is espe- cially challenging when textual and visual modali- ties are combined in a single context, e.g., an over- lay text embedded on top of an image, also known as meme. Recent efforts in multimodal representa- tion learning (Lu et al., 2019; Radford et al., 2021) pushed the boundaries of solving such problems by combining visual and textual representations of the given content. Several datasets have been pro- posed using multimodal data (Gomez et al., 2020; Kiela et al., 2020b; Sharma et al., 2020; Praman- ick et al., 2021; Suryawanshi et al., 2020; Menini …

Content truncated to 500 words