SPDB Innovation Lab at SemEval-2022 Task 3: Recognize Appropriate Taxonomic Relations Between Two Nominal Arguments with ERNIE-M Model

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 266 - 270 July 14-15, 2022 ©2022 Association for Computational Linguistics SPDB Innovation Lab at SemEval-2022 Task 3: Recognize Appropriate Taxonomic Relations Between Two Nominal Arguments with ERNIE-M Model Yue Zhou, Bowei Wei, Jianyu Liu, Yang Yang Shanghai Pudong Development Bank {zhouy93, weibw1, liujy27, yangy103}@spdb.com.cn Abstract Synonyms and antonym practices are the most common practices in our early childhood. It correlated our known words to a better place deep in our intuition. At the beginning of life for a machine, we would like to treat the ma- chine as a baby and build a similar training for it as well to present a qualified performance. In this paper, we present an ensemble model for sentence logistics classification, which out- performs the state-of-art methods. Our ap- proach essentially builds on two models includ- ing ERNIE-M and DeBERTaV3. With cross- validation and random seed tuning, we select the top performance models for the last soft en- semble and make them vote for the final answer, achieving the top 6 performance. 1 Introduction Synonym, antonym and their relations from un- structured text are fundamental problems in infor- mation classification field. These problems can be decomposed into three subtasks: word extraction using regrex, relation extraction (Zelenko et al., 2003), (Bunescu and Mooney, 2005), and classi- fying the logistics between them. However, an end-to-end model, i.e. ERNIE-M model (Ouyang et al., 2020), is proposed to solve the three tasks. Presupposed Taxonomies - Evaluating Neural- network Semantics (PreTENS) (Zamparelli et al., 2022) is a task to predict the acceptability of sim- ple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. In this paper, we first present a simple approach with the ERNIE-M model to solve the task. Although the ERNIE- M model performs unexpectedly impressive, the model has poor robustness. Hence, the additional pre-trained model is introduced to solve the ro- bustness problem. The latest model DeBERTaV3 (He et al., 2021) has outstanding performance on cross-linguistic tasks, which outperforms BERT and DeBERTa on many tasks. The proposed model consists of two parts: the basic ERNIE-M model and the pre-trained model DeBERTaV3. The De- BERTaV3 model shares the same pre-trained data with ERNIE-M called XNLI (Conneau et al., 2018), which can improve the performance and robustness as well. The DeBERTaV3 model is trained inde- pendently, which has significant improvement for English but somehow brought no improvement for other languages. Based on the above conclusion, we employ the DeBERTaV3 model for English-task only. To better understand the effectiveness of the pro- posed model, we started a bunch of analyses. The first problem is the data-set limitation. Two addi- tional datasets were imported, i.e., the translated dataset from Google translation which is trans- lated from three languages, and the XNLI dataset. However, larger datasets don’t lead to better per- formance. We compared the performance of the ERNIE-M model on four sets of data: the given data, the …

Content truncated to 500 words