Sources of Transfer in Multilingual Named Entity Recognition

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8093–8104 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8093 Sources of Transfer in Multilingual Named Entity Recognition David Mueller1,2 Nicholas Andrews2 Mark Dredze1,2 1Center for Language and Speech Processing, Johns Hopkins University 2Human Language Technology Center of Excellence, Johns Hopkins University dam,noa@jhu.edu mdredze@cs.jhu.edu Abstract Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this sim- ple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consis- tently underperforms models trained on mono- lingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and signifi- cantly outperform their monolingual counter- parts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolin- gual counterparts. We find that polyglot mod- els efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters. 1 Introduction Multilingual learning—using data from multiple languages to train a single model—can take many forms, such as adapting a model from a high- resource to low-resource language (Xie et al., 2018; Ni et al., 2017; Mayhew et al., 2017; Cot- terell and Duh, 2017; Wu and Dredze, 2019; M`arquez et al., 2003), taking advantage of bene- ficial multilingual features or datasets (Kim et al., 2012; Ehrmann et al., 2011; T¨ackstr¨om, 2012), and unsupervised representation learning (Devlin et al., 2018a). We adopt the term “Polyglot” from Tsvetkov et al. (2016) to refer to models that are trained on and applied to multiple languages. There are several advantages to training a single polyglot model across languages. Single mod- els ease production requirements; only one model need be maintained. They can be more efficient, using fewer parameters than multiple monolingual models. Additionally, they can enable multilin- gual transfer (Devlin, 2018; Wu and Dredze, 2019; Pires et al., 2019). However, a key goal of polyglot learning con- cerns producing a single model that does better on each language than a monolingual model. In the context of named entity recognition, we may expect aspects of the task to transfer across lan- guages. For example, since entity names tend to be transliterated or directly used across languages, even distant languages may see benefit from train- ing a single model, e.g. “Apple” (company) is ren- dered as such in French rather than as “Pomme.” Intuitively, the more similar and the larger the set of languages, the more we should expect to see a benefit from considering them jointly. These poly- glot models can take advantage of different sets of labeled corpora in …

Content truncated to 500 words