JUST-DEEP at SemEval-2022 Task 4: Using Deep Learning Techniques to Reveal Patronizing and Condescending Language

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 23 - 28 July 14-15, 2022 ©2022 Association for Computational Linguistics BLCU-ICALL at SemEval-2022 Task 1: Cross-Attention Multitasking Framework for Definition Modeling Cunliang Kong1, Yujie Wang2, Ruining Chong1, Liner Yang1⇤, Hengyuan Zhang1, Erhong Yang1, Yaping Huang2 1School of Information Science, Beijing Language and Culture University 2School of Computer and Information Technology, Beijing Jiaotong University cunliang.kong@outlook.com Abstract This paper describes the BLCU-ICALL sys- tem used in the SemEval-2022 Task 1 Com- paring Dictionaries and Word Embeddings, the Definition Modeling subtrack, achieving 1st on Italian, 2nd on Spanish and Russian, and 3rd on English and French. We propose a transformer-based multitasking framework to explore the task. The framework integrates multiple embedding architectures through the cross-attention mechanism, and captures the structure of glosses through a masking lan- guage model objective. Additionally, we also investigate a simple but effective model ensem- bling strategy to further improve the robust- ness. The evaluation results show the effec- tiveness of our solution. We release our code at: https://github.com/ blcuicall/SemEval2022- Task1-DM. 1 Introduction Word embeddings (Mikolov et al., 2013a; Penning- ton et al., 2014; Yogatama et al., 2015) are dense and low dimensional vectors used in many NLP tasks because they are found to be useful repre- sentations of words and often lead to better per- formance in various tasks. In recent years, large pretrained language models (PLMs), such as BERT (Devlin et al., 2019) and GPT (Petroni et al., 2019) families of models, have taken the NLP field by storm, achieving state-of-the-art performance on many tasks (Min et al., 2021). The contextual em- beddings generated by PLMs are proven to capture syntax and semantic features of words (Jawahar et al., 2019; Turton et al., 2020). But for human beings, word embeddings containing these infor- mation is still a black box and unexplainable. There have been many efforts devoted to eval- uating the word embeddings’ lexical information, such as the word similarity (Landauer and Dumais, 1997; Downey et al., 2007) and analogical relation ⇤Corresponding author: Liner Yang. ������������������������� ������������������� �������������������������� ������������������� ������������������������� ������������������� �������������� ������������������� �������������������� ����� ��������������� ����� �������� ����� ����� Figure 1: Architecture of the Cross-Attention Multitask- ing Framework. (Mikolov et al., 2013c) tasks. However, these tasks can only serve as indirect evaluation methods. In light of this, Noraset et al. (2017) proposed the task of definition modeling to evaluate whether a word embedding can be employed to generate a dictio- nary gloss. Since the gloss is a direct and explicit statement of word meaning, this task provides a more transparent view. The SemEval-2022 Task 1 Comparing Dictio- naries and Word Embeddings (Mickus et al., 2022) aims at comparing the two types of semantic de- scriptions: dictionary glosses and word embed- dings. The subtrack 1 is a definition modeling task, which requires models to generate glosses from word embeddings. The task provides data from 5 languages (English, Spanish, French, Italian, Rus- sian) as well as static, character, and contextual embeddings. Our team propose a transformer-based …

Content truncated to 500 words