HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 221 - 227 July 14-15, 2022 ©2022 Association for Computational Linguistics HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection Zheng Chu†‡, Ziqing Yang‡, Yiming Cui†‡, Zhigang Chen‡, Ming Liu† †Research Center for SCIR, Harbin Institute of Technology, Harbin, China ‡State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China †{zchu,ymcui,mliu}@ir.hit.edu.cn ‡{zhengchu,zqyang5,ymcui,zgchen}@iflytek.com Abstract The same multi-word expressions may have different meanings in different sentences. They can be mainly divided into two categories, which are literal meaning and idiomatic mean- ing. Non-contextual-based methods perform poorly on this problem, and we need contextual embedding to understand the idiomatic mean- ing of multi-word expressions correctly. We use a pre-trained language model, which can provide a context-aware sentence embedding, to detect whether multi-word expression in the sentence is idiomatic usage. 1 Introduction The goal of the SemEval-2022 Task2 (Tayyar Mad- abushi et al., 2022) SubtaskA is to detect whether a multi-word expression in a sentence is idiomatic in usage. It is a multilingual task and consists of three languages: English, Portuguese and Galician. Multi-word expressions (MWEs) are expres- sions that consist of at least two words and are syntactically or semantically specific. The seman- tics of MWEs are usually divided into two types, (i) the combination of literal meanings of each word in the phrase or (ii) inherent usages (e.g., idiomatic meaning). Understanding the semantic meaning of a sentence requires the correct identification of the MWE in the sentence. Table 1 contains one case for each of the two usages. Traditional non-contextual word embedding models, such as word2vec (Mikolov et al., 2013), perform poorly at this task. Simple superposition of non-contextually word embeddings does not cor- rectly express the semantics of idiomatic phrases. Therefore, contextual embedding models (Conneau et al., 2020; Devlin et al., 2019) are required to correctly understand the meaning of multi-word expressions in idiomatic usage. We used large-scale cross-lingual pre-trained lan- guage models, multilingual BERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020), Literal When removing a big fish from a net, it should be held in a manner that suports the girth. Idiomatic It was still a respectable finish for both Fadol and Nayre, who were ranked outside the top 500 in the world but caught some big fish along the way Table 1: Examples of idiomatic and non-idiomatic usage with a softmax classifier on top of the pre-trained LM to train a binary classification model. The training data are processed before training, and reg- ularization dropout (Liang et al., 2021), adversarial training (Miyato et al., 2017; Madry et al., 2018) are used in the training process. In addition, we observed the training data and found an interest- ing phenomenon that we can get better results by post-processing after the training using heuristic rule. 2 Background 2.1 Task Description Task 2 contains two subtasks, SubtaskA is idiom detection, and SubtaskB is similarity scoring of texts containing idioms. This article focus on Sub- taskA. SubtaskA contains two …

Content truncated to 500 words