Silp_nlp at SemEval-2023 Task 2: Cross-lingual Knowledge Transfer for Mono-lingual Learning

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1183–1189 July 13-14, 2023 ©2023 Association for Computational Linguistics silp_nlp at SemEval-2023 Task 2: Cross-lingual Knowledge Transfer for Mono-lingual Learning Sumit Singh and Uma Shanker Tiwary Indian Institute of Information Technology, Allahabad sumitrsch@gmail.com ust@iiita.ac.in Abstract Our team silp_nlp participated in SemEval2023 Task 2: MultiCoNER II. Our work made sys- tems for 11 mono-lingual tracks. For lever- aging the advantage of all track knowledge we chose transformer-based pretrained models, which have strong cross-lingual transferabil- ity. Hence our model trained in two stages, the first stage for multi-lingual learning from all tracks and the second for fine-tuning individual tracks. Our work highlights that the knowledge of all tracks can be transferred to an individual track if the baseline language model has cross- lingual features. Our system positioned itself in the top 10 for 4 tracks by scoring 0.7432 macro F1 score for the Hindi track ( 7th rank ) and 0.7322 macro F1 score for the Bangla track ( 9th rank ). 1 Introduction Multilingual Complex Named Entity Recognition) (Fetahu et al., 2023b) targets the recognition of complex named entities in multilingual and multi- domain texts, encouraging researchers to develop new approaches to extract diversified entity types. The semantic structure of data (Fetahu et al., 2023a), which is also provided by organizers for training and validation and testing purposes, has six types of coarse-grained entities: person, loca- tion, group, product, medical and creative work with more than 30 fine-grained entities of these 6 coarse-grained entities. The dataset was provided for 12 mono-lingual tracks, and, in addition, this task also provided multi-language data encourag- ing the development of more generic and adaptive systems. NER task is used in many other NLP tasks. Our team participated in 11 mono-lingual tracks ( Hindi, Bangla, English, German, Spanish, French, Italian, Portuguese, Swedish, Ukrainian, and Chinese ). Our system scores 7th rank in Hindi and is secure in the top 10 in 4 languages (Fetahu et al., 2023b). Our system leveraged the advantage of cross- lingual learning through multi-stage training sim- ilar to (Wang et al., 2022) and (He et al., 2022). For this in the first stage, we have selected Cross- lingual pretrained models which cover a group of languages of our task and we fine-tune these pre- trained language models for a combined dataset of all tracks in which languages are covered by these language models for 10 epoch. It creates a model checkpoint which has knowledge of multi- ple languages for our task. In the second stage, we fine-tuned the model checkpoints which are gen- erated in the first stage for each track individually. For the Hindi and Bangla tracks our best model was achieved by Muril large model (Khanuja et al., 2021) which is combinedly fine-tuned for three tracks (English, Bangla, Hindi) in the first stage and thereafter the model checkpoints fine-tuned for Hindi and Bangla. For the best model for all other languages, we fine-tuned xlm-r (Conneau et al., 2020) …

Content truncated to 500 words