VTCC-NER at SemEval-2023 Task 6: An Ensemble Pre-trained Language Models for Named Entity Recognition

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 415–419 July 13-14, 2023 ©2023 Association for Computational Linguistics VTCC-NER at SemEval-2023 Task 6: An Ensemble Pre-trained Language Models for Named Entity Recognition Quang-Minh Tran, Xuan-Dung Doan∗ Viettel Cyperspace Center, Viettel Group, Vietnam minhtq21,dungdx4@viettel.com.vn Abstract We propose an ensemble method that combines several pre-trained language models to enhance entity recognition in legal text. Our approach achieved a 90.9873% F1 score on the private test set, ranking 2nd on the leaderboard for Se- mEval 2023 Task 6, Subtask B - Legal Named Entities Extraction. Our code is available for further exploitation at: https://github.com/ tqgminh/SemEval2023_LegalNER_VTCC. 1 Introduction Named Entity Recognition (NER) is a key prob- lem in natural language processing (NLP) that in- volves recognizing named entities in text and clas- sifying them into specific types. Recent studies on NER have achieved many promising results. Most of state-of-the-art NER models are based on pre- trained language models, which have been trained on large text corpora and are effective for word representation. However, Domain-Specific NER, such as Legal NER, remains a challenging task. Named entities in legal texts are more different and specific than named entities in general texts. For example, a per- son’s name in legal texts may refer to a petitioner, respondent or lawyer. Therefore, some pre-trained language models for legal domain have been re- leased for use not only in Legal NER, but also for other NLP tasks in legal texts. In this work, we propose a model that combines a pre-trained language model and some techniques, such as CRF or dependency parsing, for NER in legal texts. An ensemble method is used to further boost the performance. 2 Related Works NER in legal texts has been researched for many years and in many languages, but it still remains a challenging task. Dozier et al. (2010) defined ∗* Corresponding author five legal named entities in US legal texts (judges, attorneys, companies, jurisdictions and courts) and developed a named entity recognition and resolu- tion system based on pattern-based and statistical models. Cardellino et al. (2017) trained a Named Entity Recognizer, Classifier and Linker by map- ping the LKIF ontology to the YAGO ontology and developed a structured knowledge represen- tation of legal concepts. Due to variant writing style and vocabulary across different languages and forms, some datasets have been published to create separate models, such as German LER (Leit- ner et al., 2020), LegalNERo (P˘ais, et al., 2021), LeNER-Br (Luz de Araujo et al., 2018), and Legal NER (Kalamkar et al., 2022). Applying pre-trained language models boost the accuracy of NER tasks. BERT (Devlin et al., 2018) is a language model based on Transformer archi- tecture, and was trained on two different tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). RoBERTa (Liu et al., 2019) is a robust version of BERT, and was trained on a much larger dataset and using a more effec- tive training procedure. XLM-RoBERTa (Conneau et al., 2019) is based on RoBERTa architecture …

Content truncated to 500 words