MultiCoNER: A Large-scale Multilingual Dataset for Complex Named Entity Recognition

Proceedings of the 29th International Conference on Computational Linguistics, pages 3798–3809 October 12–17, 2022. 3798 MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition Shervin Malmasi, Anjie Fang∗, Besnik Fetahu∗, Sudipta Kar∗, Oleg Rokhlenko Amazon.com, Seattle, WA, USA {malmasi,njfn,besnikf,sudipkar,olegro}@amazon.com Abstract We present MULTICONER, a large multilin- gual dataset for Named Entity Recognition that covers 3 domains (Wiki sentences, ques- tions, and search queries) across 11 languages, as well as multilingual and code-mixing sub- sets. This dataset is designed to represent contemporary challenges in NER, including low-context scenarios (short and uncased text), syntactically complex entities like movie titles, and long-tail entity distributions. The 26M to- ken dataset is compiled from public resources using techniques such as heuristic-based sen- tence sampling, template extraction and slot- ting, and machine translation. We applied two NER models on our dataset: a baseline XLM-RoBERTa model, and a state-of-the- art GEMNET model that leverages gazetteers. The baseline achieves moderate performance (macro-F1=54%), highlighting the difficulty of our data. GEMNET, which uses gazetteers, improvement significantly (average improve- ment of macro-F1=+30%). MULTICONER poses challenges even for large pre-trained language models, and we believe that it can help further research in building robust NER systems. MULTICONER is publicly available,1 and we hope that this resource will help advance re- search in various aspects of NER. 1 Introduction Named Entity Recognition (NER) is a core task in Natural Language Processing which involves identifying entities in text, and recognizing their types (e.g., classifying entities as a person or loca- tion). Recently, the development of Transformer- based NER approach have results in new state- of-the-art (SOTA) results on well-known bench- mark datasets like CoNLL03 and OntoNotes (De- vlin et al., 2019). Despite these strong results, ∗These authors contributed equally in this work. 1https://registry.opendata.aws/multiconer/ Figure 1: Some examples for all the languages incor- porated in MULTICONER. there remain a number of practical challenges that may not be represented by these existing datasets. As noted by Augenstein et al. (2017), increas- ingly higher scores on these datasets are driven by several factors: • Well-formed data, with punctuation and capital- ized nouns, makes the NER task easier, provid- ing the model with additional contextual cues. • Texts from articles and the news domain contain long sentences with rich context around entities, providing valuable signals about the boundaries and types of entities. • Data from the news domain2 contains “easy” en- tities such as country, city, and person names, al- lowing pre-trained models to perform well due to their existing knowledge of such tokens. • Memorization effects, due to large overlap of entities between the train and test sets also in- creases performance. Accordingly, models trained on existing bench- mark datasets such as CoNLL03 tend to perform significantly worse on unseen entities or noisy text (Meng et al., 2021). 2e.g. CoNLL03 (Sang and De Meulder, 2003) 3799 1.1 Contemporary Challenges in NER There are many challenging scenarios for NER outside of the news domain. We categorize the challenges typically encountered in NER ac- cording to several dimensions: …

Content truncated to 500 words