LingJing at SemEval-2022 Task 3: Applying DeBERTa to Lexical-level Presupposed Relation Taxonomy with Knowledge Transfer

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 239 - 246 July 14-15, 2022 ©2022 Association for Computational Linguistics LingJing at SemEval-2022 Task 3: Applying DeBERTa to Lexical-level Presupposed Relation Taxonomy with Knowledge Transfer Fei Xia1,2∗, Bin Li3∗, Yixuan Weng1∗, Shizhu He1,2, Bin Sun3, Shutao Li3, Kang Liu1,2, Jun Zhao1,2 1 National Laboratory of Pattern Recognition Institute of Automation, CAS 2 School of Artificial Intelligence, University of Chinese Academy of Sciences 3 College of Electrical and Information Engineering, Hunan University {libincn, shutao_li, sunbin611}@hnu.edu.cn, wengsyx@gmail.com, xiafei2020@ia.ac.cn, {shizhu.he, kliu, jzhao}@nlpr.ia.ac.cn Abstract This paper presents the results and main find- ings of our system on SemEval-2022 Task 3 Presupposed Taxonomies: Evaluating Neu- ral Network Semantics (PreTENS)1. This task aims at semantic competence with specific at- tention on the evaluation of language mod- els, which is a task with respect to the recog- nition of appropriate taxonomic relations be- tween two nominal arguments. Two sub-tasks including binary classification and regression are designed for the evaluation. For the clas- sification sub-task, we adopt the DeBERTa- v3 pre-trained model for fine-tuning datasets of different languages. Due to the small size of the training datasets of the regression sub- task, we transfer the knowledge of classifica- tion model (i.e., model parameters) to the re- gression task. The experimental results show that the proposed method achieves the best results on both sub-tasks. Meanwhile, we also report negative results of multiple training strategies for further discussion. All the ex- perimental codes are open-sourced at https: //github.com/WENGSYX/Semeval. 1 Introduction In order to dive into the capability of the current language models (Bengio et al., 2000; Howard and Ruder, 2018), we take part in and apply the pre-training language model on the SemEval-2022 Tasks 3: the Presupposed Taxonomies: Evaluating Neural Network Semantics (PreTENS) (Zamparelli et al., 2022). To evaluate the model performance comprehensively, two sub-tasks are designed in this PreTENS task: Sub-task 1) Binary classification task2, which consists in predicting the acceptability label as- signed to each sentence of the test set. For example, “I like trees, and in particular birches” is acceptable ∗These authors contribute equally to this work. 1https://sites.google.com/view/semeval2022-pretens/ 2https://codalab.lisn.upsaclay.fr/competitions/1292 while “ I like oaks, and in particular trees” is unac- ceptable, so they are labeled 1 and 0, respectively. Sub-task 2) Regression sub-task3, which con- sists in predicting the average score assigned by human annotators on a seven-point Likert scale (Joshi et al., 2015) with respect to the subset of data evaluated via crowdsourcing. For example,“I like governors, an interesting type of politician” is more acceptable than “I like politicians, an interest- ing type of farmer”, so the former will also have a higher score (6.16) than the latter (1.42). It is noted that both sub-tasks comprise datasets in 3 languages: English, Italian, French, where French and Italian are slightly adapted translations of the English dataset. For each sub-task, every sample is formed as the arguments A and B, e.g., comparatives (I like A more than B), exemplifica- tions (I like A, and in particular …

Content truncated to 500 words