SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER)

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1412 - 1437 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER) Shervin Malmasi, Anjie Fang∗, Besnik Fetahu∗, Sudipta Kar∗, Oleg Rokhlenko Amazon.com, Inc. Seattle, WA, USA {malmasi,njfn,besnikf,sudipkar,olegro}@amazon.com Abstract We present the findings of SemEval-2022 Task 11 on Multilingual Complex Named Entity Recognition MULTICONER.1 Divided into 13 tracks, the task focused on methods to identify complex named entities (like media titles, prod- ucts, and groups) in 11 languages in both mono- lingual and multi-lingual scenarios. Eleven tracks were for building monolingual NER models for individual languages, one track fo- cused on multilingual models able to work on all languages, and the last track featured code-mixed texts within any of these languages. The task used the MULTICONER dataset, com- posed of 2.3 million instances in Bangla, Chi- nese, Dutch, English, Farsi, German, Hindi, Ko- rean, Russian, Spanish, and Turkish. Results showed that methods fusing external knowl- edge into transformer models achieved the best performance. The largest gains were on the Cre- ative Work and Group entity classes, which are still challenging even with external knowledge. MULTICONER was one of the most popular tasks in SemEval-2022 and it attracted 377 par- ticipants during the practice phase. The final test phase had 236 participants, and 55 teams submitted their systems. 1 Introduction Processing complex and ambiguous Named En- tities (NEs) is a challenging NLP task in practi- cal and open-domain settings but has not received sufficient attention from the research community. Complex NEs, like the titles of creative works (movie/book/song/software names) are not simple nouns and are harder to recognize (Ashwini and Choi, 2014). They can take the form of any lin- guistic constituent, like an imperative clause (“Dial M for Murder”), and do not look like traditional NEs (Person names, locations, organizations). This ambiguity makes it challenging to recognize them ∗These authors contributed equally to this work. 1https://multiconer.github.io/ based on their context. Such titles can also be se- mantically ambiguous, e.g. “On the Beach” can be a preposition or refer to a movie.2 Finally, such entities usually grow at a faster rate than traditional categories, and emerging entities pose yet another challenge. Neural models (e.g. Transformers) have pro- duced high scores on benchmark datasets like CoNLL03/OntoNotes (Devlin et al., 2018). How- ever, as noted by Augenstein et al. (2017), these scores are driven by the use of well-formed news text, the presence of “easy” entities (e.g. person names), and memorization due to entity overlap between train/test sets; these models perform sig- nificantly worse on complex/unseen entities (Meng et al., 2021; Fetahu et al., 2021). Researchers us- ing NER on downstream tasks have also noted that a significant proportion of their errors are due to NER systems failing to recognize complex enti- ties (Luken et al., 2018; Hanselowski et al., 2018). Examples of such challenges are highlighted in Table 1. For this task, we created the MULTICONER dataset (Malmasi et al., 2022) to address …

Content truncated to 500 words