Extending Multilingual BERT to Low-Resource Languages

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2649 Extending Multilingual BERT to Low-Resource Languages Zihan Wang∗ University of Illinois Urbana-Champaign Urbana, IL 61801, USA zihanw2@illinois.edu Karthikeyan K∗ Indian Institute of Technology Kanpur Kanpur, Uttar Pradesh 208016, India kkarthi@cse.iitk.ac.in Stephen Mayhew† Duolingo Pittsburgh, PA, 15206, USA stephen@duolingo.com Dan Roth University of Pennsylvania Philadelphia, PA 19104, USA danroth@seas.upenn.edu Abstract Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. However, this success is focused only on the top 104 lan- guages in Wikipedia it was trained on. In this paper, we propose a simple but effective ap- proach to extend M-BERT (E-MBERT) so it can benefit any new language, and show that our approach aids languages that are already in M-BERT as well. We perform an extensive set of experiments with Named Entity Recogni- tion (NER) on 27 languages, only 16 of which are in M-BERT, and show an average increase of about 6% F1 on M-BERT languages and 23% F1 increase on new languages. We re- lease models and code at 1. 1 Introduction Recent works (Wu and Dredze, 2019; K et al., 2020) have shown the zero-shot cross-lingual abil- ity of M-BERT (Devlin et al., 2018) on various semantic and syntactic tasks – just fine-tuning on English data allows the model to perform well on other languages. Cross-lingual learning is imper- ative for low-resource languages such as Somali and Uyghur, as obtaining supervised training data in these languages is particularly hard. However, M-BERT is not pre-trained with these languages, thus limiting its performance on them. Languages like Oromo, Hausa, Amharic and Akan are spoken by more than 20 million people, yet M-BERT does not cover them. Indeed, there are about 40002 writ- ∗Equal Contribution; most of this work was done while the authors interned at the University of Pennsylvania. †This work was done while the author was a student at the University of Pennsylvania. 1http://cogcomp.org/page/publication_ view/912 2https://www.ethnologue. com/enterprise-faq/ how-many-languages-world-are-unwritten-0 0 10 20 30 40 50 60 70 80 In M-BERT Out of M-BERT Performance (F1) Average performance on NER M-BERT_supervised M-BERT zeroshot E-MBERT zeroshot Figure 1: Comparison between M-BERT and our proposed approach E-MBERT: We report average zero-shot NER performance on 16 languages that are al- ready in M-BERT and 11 new language that are out of M-BERT; M-BERT performance with supervised NER data is also reported as an upper-bound. In both lan- guages in M-BERT and out of M-BERT, our method E-MBERT performs better than M-BERT. ten languages, of which M-BERT covers only the top 104 languages (less than 3%). One straightforward way to extend the notion of M-BERT to languages not covered by it is to train a new M-BERT from scratch to include the new lan- guage. However, this is extremely time-consuming and expensive: training BERT-base takes about four days with four cloud TPUs (Devlin et al., 2019). Alternatively, one can train a BERT with two languages, …

Content truncated to 500 words