Massively Multilingual Transfer for NER

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 151 Massively Multilingual Transfer for NER Afshin Rahimi∗ Yuan Li∗ Trevor Cohn School of Computing and Information Systems The University of Melbourne yuanl4@student.unimelb.edu.au {rahimia,t.cohn}@unimelb.edu.au Abstract In cross-lingual transfer, NLP models over one or more source languages are applied to a low- resource target language. While most prior work has used a single source model or a few carefully selected models, here we con- sider a “massive” setting with many such mod- els. This setting raises the problem of poor transfer, particularly from distant languages. We propose two techniques for modulating the transfer, suitable for zero-shot or few-shot learning, respectively. Evaluating on named entity recognition, we show that our tech- niques are much more effective than strong baselines, including standard ensembling, and our unsupervised method rivals oracle selec- tion of the single best individual model.1 1 Introduction Supervised learning remains king in natural lan- guage processing, with most tasks requiring large quantities of annotated corpora. The majority of the world’s 6,000+ languages however have lim- ited or no annotated text, and therefore much of the progress in NLP has yet to be realised widely. Cross-lingual transfer learning is a tech- nique which can compensate for the dearth of data, by transferring knowledge from high- to low- resource languages, which has typically taken the form of annotation projection over parallel corpora or other multilingual resources (Yarowsky et al., 2001; Hwa et al., 2005), or making use of trans- ferable representations, such as phonetic transcrip- tions (Bharadwaj et al., 2016), closely related lan- guages (Cotterell and Duh, 2017) or bilingual dic- tionaries (Mayhew et al., 2017; Xie et al., 2018). Most methods proposed for cross-lingual trans- fer rely on a single source language, which lim- its the transferable knowledge to only one source. ∗Both authors contributed equally to this work. 1The code and the datasets will be made available at https://github.com/afshinrahimi/mmner. The target language might be similar to many source languages, on the grounds of the script, word order, loan words etc, and transfer would benefit from these diverse sources of information. There are a few exceptions, which use transfer from several languages, ranging from multitask learning (Duong et al., 2015; Ammar et al., 2016; Fang and Cohn, 2017), and annotation projection from several languages (T¨ackstr¨om, 2012; Fang and Cohn, 2016; Plank and Agi´c, 2018). How- ever, to the best of our knowledge, none of these approaches adequately account for the quality of transfer, but rather “weight” the contribution of each language uniformly. In this paper, we propose a novel method for zero-shot multilingual transfer, inspired by re- search in truth inference in crowd-sourcing, a re- lated problem, in which the ‘ground truth’ must be inferred from the outputs of several unreliable an- notators (Dawid and Skene, 1979). In this prob- lem, the best approaches estimate each model’s reliability, and their patterns of mistakes (Kim and …

Content truncated to 500 words