HW-TSC at SemEval-2022 Task 3: A Unified Approach Fine-tuned on Multilingual Pretrained Model for PreTENS

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 291 - 297 July 14-15, 2022 ©2022 Association for Computational Linguistics HW-TSC at SemEval-2022 Task 3: A Unified Approach Fine-tuned on Multilingual Pretrained Model for PreTENS Yinglu Li, Min Zhang, Xiaosong Qiao, Minghan Wang Hao Yang, Shimin Tao, Ying Qin Huawei Translation Services Center, Beijing, China {liyinglu, zhangmin186, qiaoxiaosong, wangminghan, yanghao30, taoshimin, qinying}@huawei.com Abstract In the paper, we describe a unified system for task 3 of SemEval-2022. The task aims to rec- ognize the semantic structures of sentences by providing two nominal arguments and to eval- uate the degree of taxonomic relations. We utilise the strategy that adding language pre- fix tag in the training set, which is effective for the model. We split the training set to avoid the translation information to be learnt by the model. For the task, we propose a uni- fied model fine-tuned on the multilingual pre- trained model, XLM-RoBERTa. The model performs well in subtask 1 (the binary classi- fication subtask). In order to verify whether our model could also perform better in subtask 2 (the regression subtask), the ranking score is transformed into classification labels by an up-sampling strategy. With the ensemble strat- egy, the performance of our model can also be improved. As a result, the model obtained the second place for subtask 1 and subtask 2 in the competition evaluation. 1 Introduction As we all know, the proposal of BERT(Devlin et al., 2018; Vaswani et al., 2017), which is based on masked language modeling, is a huge milestone in the history of natural language understanding (Peters et al., 2018; Schuster et al., 2019). Com- pared with various language representation mod- els, BERT successfully pushes the GLUE score at 7.7 points absolute improvement(Devlin et al., 2018). Soon, different types of language models such as XLNet(You et al., 2019), RoBERTa(Liu et al., 2019),mBert (Devlin et al., 2018; Radford and Narasimhan, 2018) and XLM (Lample and Conneau, 2019) are also proposed. Compared with some strong monolingual models introduced above, XLM-RoBERTa is more competitive on the GLUE and XNLI benchmarks(Conneau et al., 2019). There are lots of pretrained models pro- posed in recent years, which are capable of learning the implicit knowledge. Some works have proved that the neural language model has learnt the im- plicit linguistic knowledge and this knowledge can significantly affect the predictions through fine- tuning(Miaschi et al., 2020; Puccetti et al., 2021). It has been believed that the pretrained models trained on Wikipedia and other datasets already have some implicit knowledge. Implicit knowledge could be classified into two categories: the connection be- tween two objects, and the implicit logic and syntax behind the sentence. It means, we need to build a model which is capable of verifying the rational- ity and reliability of sentences, testing whether the latent knowledge can be expressed explicitly. Presupposed taxonomy is a kind of concept in computational linguistics. Two arguments could have several different taxonomy relationships. For example, the sentence "I like piano, but not …

Content truncated to 500 words