Mitra Behzadi at SemEval-2022 Task 5 : Multimedia Automatic Misogyny Identification method based on CLIP

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 82 - 87 July 14-15, 2022 ©2022 Association for Computational Linguistics RIGA at SemEval-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling Eduards Mukans mukans.work@gmail.com University of Latvia Gus Strazds gs15014@students.lu.lv University of Latvia Guntis Barzdins guntis@latnet.lv University of Latvia, IMCS Abstract Described are our two entries "emukans" and "guntis" for the definition modeling track of CODWOE SemEval-2022 Task 1. Our ap- proach is based on careful scaling of a GRU re- current neural network, which exhibits double descent of errors, corresponding to significant improvements also per human judgement. Our results are in the middle of the ranking table per official automatic metrics. 1 Introduction The definition modeling track of SemEval-2022 Task 1: CODWOE - COmparing Dictionaries and WOrd Embeddings (Mickus et al., 2022) challenged participants to generate dictionary glosses from individual word embedding vectors. This paper describes two CODWOE submissions, "emukans" and "guntis", where the first focuses on the automatic CODWOE scores, but the second attempts to gauge the relationships between scaling laws, the automated metrics, and human evaluation. Our submissions achieved competitive results (see Figure 3) on the MoverScore official metric - scor- ing 1st for French, 2nd for Spanish, and 3rd for Russian. Our approach was to apply classical recur- rent networks, such as Long Short-term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014), to definition modeling and investigate how model scaling impacts performance. The scaling effect is well investigated for transformers, but not so much for RNNs. Recently, the main focus in deep learning has skewed from searching for new model architectures to investigating how various factors impact the training process and overall system per- formance (Nakkiran et al., 2019; Kaplan et al., 2020; Gordon et al., 2021). The main factors are: the amount of data, the amount of compute, and the size of the model (parameter count). In the competition the data amount is fixed and no use of external data is permitted, thus we investigate how scaling model size and training time impacts train- ing progress and model performance for recurrent models. In our experiments we did observe deep double descent effects: epoch-wise double descent with respect to both cross-entropy loss and prediction accuracy on a validation data set, more pronounced with increasing model size. We also investigated the automatic metrics used for evaluating submissions and their correlation with human evaluation, focusing primarily on the MoverScore metric (Wei Zhao, 2019). MoverScore does correlate with human evaluation, but not nec- essarily very strong, at least for this dataset. We find that the double descent effect seen with re- spect to prediction accuracy can also be observed for MoverScore. 2 Background The CODWOE shared task invites participants to compare two types of semantic descriptions: dictio- nary glosses and word embedding representations. The task consists of 2 subtracks: definition model- ing and reverse dictionary. In definition modeling participants have to generate glosses from word embedding representations. The …

Content truncated to 500 words