ZhichunRoad at SemEval-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 197 - 203 July 14-15, 2022 ©2022 Association for Computational Linguistics ZhichunRoad at SemEval-2022 Task 2: Adversarial Training and Contrastive Learning for Multiword Representations Xuange Cui , Wei Xiong , Songlin Wang JD.com, Beijing, China {cuixuange,xiongwei9,wangsonglin3}@jd.com Abstract This paper presents our contribution to the SemEval-2022 Task 2: Multilingual Idiomatic- ity Detection and Sentence Embedding. We explore the impact of three different pre-trained multilingual language models in the SubTaskA. By enhancing the model generalization and ro- bustness, we use the exponential moving aver- age (EMA) method and the adversarial attack strategy. In SubTaskB, we add an effective cross-attention module for modeling the rela- tionships of two sentences. We jointly train the model with a contrastive learning objective and employ a momentum contrast to enlarge the number of negative pairs. Additionally, we use the alignment and uniformity properties to measure the quality of sentence embeddings. Our approach obtained competitive results in both subtasks. 1 Introduction In recent years, the pre-trained models have been widely used and play a vital role in the natural lan- guage processing tasks. The success of language models relies on huge amounts of unlabeled data and the useful representation layers that are de- signed to draw on information from the surround- ing context (Devlin et al., 2018;Nedumpozhimana and Kelleher, 2021). However, more recent studies show that even state-of-the-art pre-trained contex- tual models (e.g. BERT) can’t accurately represent idiomatic expressions (Yu and Ettinger, 2020;Gar- cia et al., 2021). One reason for this is that many expressions can be used both literally and idiomati- cally. Specifically, the size of vocabulary can’t in- crease indefinitely, which makes representing id- iomatic phrases particularly challenging (Shwartz, 2021;Tayyar Madabushi et al., 2021). Idioms occur in almost all languages, to distinguish whether an expression has an idiomatic sense would leverage both cross-lingual models and multiword expres- SubTask Train Dev Test Desc zero-shot 4492 740 763 binary one-shot 4492 740 763 classification pre-train 24498 2000 3827 semantic fine-tune 6573 2182 2263 similarity Table 1: The statistics of datasets. sions (MWEs). The SemEval 2022 Task 2(Tay- yar Madabushi et al., 2022) is aimed at detecting and representing MWEs and presents a novel mul- tilingual dataset across English, Portuguese and Galician. And this task consists of two different subtasks to evaluate the model’s ability to identify and capture idiomaticity. Our contributions can be summarized as fol- lows: 1) We choose three transformer-based lan- guage models from the XTREME LeaderBoard (Hu et al., 2020), and compare the effective- ness of mBERTbase (Devlin et al., 2018), XLM- Rbase(Conneau et al., 2020), and InfoXLMbase(Chi et al., 2021). 2) We adopt the exponential moving average method (EMA) and adversarial training strategy to improve the model’s generalization and robustness. We achieved considerably performance gain of 2.91%, 3.87% over the baseline solution, and ranked 12th in the zero-shot settings and 4th in the one-shot settings. 3) With finetuning on the supervised target datasets, we use cross-attention module and jointly train the model with …

Content truncated to 500 words