SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2247–2265 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2) Besnik Fetahu Sudipta Kar Zhiyu Chen Oleg Rokhlenko Shervin Malmasi Amazon.com, Inc. Seattle, WA, USA {besnikf,sudipkar,zhiyuche,olegro,malmasi}@amazon.com Abstract We present the findings of SemEval-2023 Task 2 on Fine-grained Multilingual Named En- tity Recognition (MULTICONER 2).1 Divided into 13 tracks, the task focused on meth- ods to identify complex fine-grained named entities (like WRITTENWORK, VEHICLE, MUSICALGRP) across 12 languages, in both monolingual and multilingual scenarios, as well as noisy settings. The task used the MULTICONER V2 dataset, composed of 2.2 million instances in Bangla, Chinese, English, Farsi, French, German, Hindi, Italian., Por- tuguese, Spanish, Swedish, and Ukrainian. MULTICONER 2 was one of the most popu- lar tasks of SemEval-2023. It attracted 842 submissions from 47 teams, and 34 teams sub- mitted system papers. Results showed that com- plex entity types such as media titles and prod- uct names were the most challenging. Meth- ods fusing external knowledge into transformer models achieved the best performance, and the largest gains were on the Creative Work and Group classes, which are still challenging even with external knowledge. Some fine-grained classes proved to be more challenging than others, such as SCIENTIST, ARTWORK, and PRIVATECORP. We also observed that noisy data has a significant impact on model perfor- mance, with an average drop of 10% on the noisy subset. The task highlights the need for future research on improving NER robustness on noisy data containing complex entities. 1 Introduction Complex Named Entities (NE), like the titles of creative works, are not simple nouns and pose chal- lenges for NER systems (Ashwini and Choi, 2014). They can take the form of any linguistic constituent, like an imperative clause (“Dial M for Murder”), and do not look like traditional NEs (Persons, lo- cations, etc.). This syntactic ambiguity makes it challenging to recognize them based on context. 1https://multiconer.github.io We organized the Multilingual Complex NER (MultiCoNER) task (Malmasi et al., 2022b) at SemEval-2022 to address these challenges in 11 languages, receiving a positive community re- sponse with 34 system papers. Results confirmed the challenges of processing complex and long- tail NEs: even the largest pretrained Transformers did not achieve top performance without external knowledge. The top systems infused transformers with knowledge bases and gazetteers. However, such solutions are brittle against out of knowledge- base entities and noisy scenarios (e.g. spelling mistakes and typos). For entities with fine-grained classes, apart from the entity surface form, the con- text is critical in determining the correct class. MULTICONER 2 expanded on these challenges by adding fine-grained NER classes, and the in- clusion of noisy input. Fine-grained NER requires models to distinguish between sub-types of enti- ties that differ only at the fine-grained level, e.g. SCIENTIST vs. ATHLETE. In these cases, it is crucial for models to capture the entity’s context. In terms of noise, we assessed how small pertur- bations in the entity …

Content truncated to 500 words