RIT Boston at SemEval-2022 Task 5: Multimedia Misogyny Detection By Using Coherent Visual and Language Features from CLIP Model and Data-centric AI Principle

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 636 - 641 July 14-15, 2022 ©2022 Association for Computational Linguistics RIT Boston at SemEval-2022 Task 5: Multimedia Misogyny Detection By Using Coherent Visual and Language Features from CLIP Model and Data-centric AI Principle Lei Chen∗ Rakuten Institute of Technology (RIT) Boston, MA lei.a.chen@rakuten.com Houwei Chou∗ RIT houwei.chou@rakuten.com Abstract Detecting MEME images to be misogynous or not is an application useful on curbing online hateful information against women. In the SemEval-2022 Multimedia Automatic Misogyny Identification (MAMI) challenge, we designed a system using two simple but effective principles. First, we leverage on recently emerging Transformer models pre- trained (mostly in a self-supervised learning way) on massive data sets to obtain very effec- tive visual (V) and language (L) features. In particular, we used the CLIP (Radford et al., 2021) model provided by OpenAI to obtain coherent V and L features and then simply used a logistic regression model to make binary predictions. Second, we emphasized more on data rather than tweaking models by following the data-centric AI principle. These principles were proven to be useful and our final macro-F1 is 0.778 for the MAMI task A and ranked the third place among participant teams. 1 Introduction Systematic inequality and discrimination to women does not appear offline but also in online communi- cation. MEME is an image characterized by a vi- sual content with an overlaying text added MEME creators. Although most of MEMEs are created with the intention of making funny jokes, some of MEMEs are created as a form against women. Therefore, identifying misogynous MEMEs is im- portant for curbing such misuse. In the SemEval-2022, the task 5, Multimedia Au- tomatic Misogyny Identification (MAMI) (Fersini et al., 2022), was organized for this purpose. The challenge consists of two sub-tasks, including the task A, which is determining a MEME be misog- ynous or not, and the task B which distinguishes non-misogynous and 4 overlapped misogyny sub- types. ∗Equal contributor We participated in the MAMI challenge and fo- cused on the task A. Our solutions focused on ex- ploring various pre-trained Transformer models for extracting textual and visual features and utilizing a simple logistic regression (LR) model to make binary predictions. In addition, following a new trend in AI research, which is relying on the power provided by data more, i.e., data-centric AI (Ng, 2021b,a), we expanded available training data by manually marking more samples from the evalu- ation set. As a result, by jointly utilizing these methods, our team ended up on obtaining the third rank in the task A. 2 Related work Automatic Misogyny Identification (AMI) has be- come an active research topic in natural language processing (NLP). For example, in the IberEval- 2018, the AMI task was introduced as a new task (Fersini et al., 2018). The task consists of the three sub-tasks, i.e., misogyny identification, misogynis- tic behavior categorization, and target classifica- tion. The misogyny related annotations are made on both Spanish and English tweets. Among 11 …

Content truncated to 500 words