Dynamic Gazetteer Integration in Multilingual Models for Cross-Lingual and Cross-Domain Named Entity Recognition

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2777 - 2790 July 10-15, 2022 ©2022 Association for Computational Linguistics Dynamic Gazetteer Integration in Multilingual Models for Cross-Lingual and Cross-Domain Named Entity Recognition Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, Shervin Malmasi Amazon, USA {besnikf, njfn, olegro, malmasi}@amazon.com Abstract Named entity recognition (NER) in a real- world setting remains challenging and is im- pacted by factors like text genre, corpus qual- ity, and data availability. NER models trained on CoNLL do not transfer well to other do- mains, even within the same language. This is especially the case for multi-lingual models when applied to low-resource languages, and is mainly due to missing entity information. We propose an approach that with limited ef- fort and data, addresses the NER knowledge gap across languages and domains. Our novel approach uses a token-level gating layer to augment pre-trained multilingual transformers with gazetteers containing named entities (NE) from a target language or domain. This ap- proach provides the flexibility to jointly in- tegrate both textual and gazetteer informa- tion dynamically: entity knowledge from gazetteers is used only when a token’s textual representation is insufficient for the NER task. Evaluation on several languages and domains demonstrates: (i) a high mismatch of reported NER performance on CoNLL vs. domain spe- cific datasets, (ii) gazetteers significantly im- prove NER performance across languages and domains, and (iii) gazetteers can be flexibly in- corporated to guide knowledge transfer. On cross-lingual transfer we achieve an improve- ment over the baseline with F1=+17.6%, and with F1=+21.3% for cross-domain transfer. 1 Introduction Advances in pre-trained models have achieved state of the art results for NER (Conneau et al., 2020; Yamada et al., 2020). Models like XLM-RoBERTa (XLMR) (Conneau et al., 2020) offer advantages as they can be applied on several languages with little fine-tuning to obtain optimal NER performance, with an F1 score of 92.92 for English and an aver- age of 89.43 across all languages in CoNLL (Sang and Meulder, 2003). Source Lang (EN) Target Lang (DE) estonia held the presidency of the council of the european union in the second half of 2017 zum 1. juli 2017 über- nahm estland zum ersten mal seit seinem beitritt die eu-rat- spräsidentschaft. Source Domain (news) Target Domain (Q&A) the tradition continued with fig- ures including james s. skinner. when did dear prudence come out? Table 1: Example snippets in multiple-languages and domains. NER needs to resolve equivalent NE surface forms across languages, e.g. “Presidency of the Euro- pean Council” to “EU-Ratspresidäntschaft”, or across domains where entity distribution change (second row, where entity types are marked in different colors). While NER results obtained on CoNLL have reached remarkable levels, in real-world settings, NER faces many challenges, related to application domain, language, or data quality. For uses cases such as Web search queries or utterances coming from voice assistants, data quality and obtaining annotations are an issue. Such corpora usually have low context and no …

Content truncated to 500 words