AaltoNLP at SemEval-2022 Task 11: Ensembling Task-adaptive Pretrained Transformers for Multilingual Complex NER

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1477 - 1482 July 14-15, 2022 ©2022 Association for Computational Linguistics AaltoNLP at SemEval-2022 Task 11: Ensembling Task-adaptive Pretrained Transformers for Multilingual Complex NER Aapo Pietiläinen Shaoxiong Ji Aalto University {aapo.pietilainen; shaoxiong.ji}@aalto.fi Abstract This paper presents the system description of team AaltoNLP for SemEval-2022 shared task 11: MultiCoNER. Transformer-based models have produced high scores on standard Named Entity Recognition (NER) tasks. However, accuracy on complex named entities is still low. Complex and ambiguous named entities have been identified as a major error source in NER tasks. The shared task is about multi- lingual complex named entity recognition. In this paper, we describe an ensemble approach, which increases accuracy across all tested lan- guages. The system ensembles output from multiple same architecture task-adaptive pre- trained transformers trained with different ran- dom seeds. We notice a large discrepancy be- tween performance on development and test data. Model selection based on limited devel- opment data may not yield optimal results on large test data sets. 1 Introduction SemEval-2022 shared task 11: MultiCoNER (Mal- masi et al., 2022b) was about complex Named En- tity Recognition (NER) in the multilingual context. Transformer-based models such as BERT (Devlin et al., 2019) have reached high accuracy in standard NER tasks. However, the models still struggle with complex and code-mixed named entities (Meng et al., 2021; Fetahu et al., 2021). The shared task tries to address the challenges regarding complex NER in a multilingual context. Out of the 13 avail- able tracks, we focus on 5 monolingual tracks. We build models for Bangla, English, Farsi, German, and Korean. Our strategy is to build an ensemble approach to increase accuracy compared to the baseline model. To tackle multilingualism, we build an approach that starts from the same XLM-RoBERTa (Con- neau et al., 2019) encoder and after fine-tuning, achieves good performance across languages. We propose two ensembling approaches: (1) naive en- sembling and (2) end-to-end (E2E) ensembling. Naive ensembling is a test-time ensembling where class scores from individually trained models are added together. E2E ensembling trains two encoder models jointly by concatenating their results and passing the concatenated predictions to the final layer. In addition to these strategies, we adapt the encoder model to data context using continued pre- training (Gururangan et al., 2020). We discover that naive ensembling produces good results with few models and by just using different random seeds for training. We also dis- cover that performance can drastically vary be- tween data sets and model selection based on de- velopment data may not yield good results on large test sets. Our final rankings are 11th for Farsi, 12th for Bangla, 13th for German, 15th for Korean, and 25th for English. For each language, our ranking is on the lower half of the participants. The code for our submission has been released 1. 2 Background In NER, complex and ambiguous entities are hard to classify correctly. Out of the 13 available tracks, we participated in the …

Content truncated to 500 words