SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1593 - 1596 July 14-15, 2022 ©2022 Association for Computational Linguistics SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models Changyu Hou1, Jun Wang1∗, Yixuan Qiao1, Peng Jiang1, Peng Gao1, Guotong Xie1∗ 1 Ping An Healthcare Technology, Beijing, China junwang.deeplearning@gmail.com Qizhi Lin2, Xiaopeng Wang2, Xiandi Jiang2, Benqi Wang2, Qifeng Xiao2 2 Ping An Puhui Enterprise Management Co. Ltd, Shanghai, China wangxiaopeng069@lu.com Abstract Large scale pre-training models have been widely used in named entity recognition (NER) tasks. However, model ensemble through pa- rameter averaging or voting can not give full play to the differentiation advantages of dif- ferent models, especially in the open domain. This paper describes our NER system in the SemEval 2022 task11: MultiCoNER. We pro- posed an effective system to adaptively ensem- ble pre-trained language models by a Trans- former layer. By assigning different weights to each model for different inputs, we adopted the Transformer layer to integrate the advantages of diverse models effectively. Experimental re- sults show that our method achieves superior performances in Farsi and Dutch. 1 Introduction NER is an essential tool in the application fields of information extraction, question answering sys- tem, syntactic analysis, machine translation, etc. It plays a vital role in the process of the practical ap- plication of natural language processing technology (Devlin et al., 2018; Meng et al., 2021). However, processing complex and ambiguous Named Enti- ties (NEs) is a challenging NLP task in practical and open-domain settings, but has not received suf- ficient attention from the research community. Complex NEs, like the titles of creative works (movie/book/song/software names), are not sim- ple nouns and are harder to recognize (Ashwini and Choi, 2014; Fetahu et al., 2021). They can take the form of any linguistic constituent, like an imperative clause (“Dial M for Murder”), and do not look like traditional NEs (Person names, lo- cations, organizations). This syntactic ambiguity makes it challenging to recognize them based on their context. Such titles can also be semantically ambiguous, e.g., “On the Beach” can be a prepo- sition or refer to a movie. Finally, such entities ∗Corresponding Author. usually grow faster than traditional categories, and emerging entities pose yet another challenge. In recent years, the deep learning methods have achieved great success in NLP (Natural Language Processing). However, some limits need to be addressed. Challenges for deep learning in NLP mainly arise from the scarcity of labeled data. One way to alleviate the need for large labeled datasets is to pre-train a model on unlabeled data via self- supervised learning, and then transfer the learned model to downstream tasks (Devlin et al., 2018). These methods have been widely applied and have made a massive breakthroughs, such as BERT (De- vlin et al., 2018), XLNET (Yang et al., 2019) and GPT (Radford et al., 2018). Pre-trained neural models (e.g., Transformers) have produced high scores on benchmark datasets like CoNLL03/OntoNotes (Devlin et al., 2018). However, as noted by Augenstein …

Content truncated to 500 words