UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10186 UNKs Everywhere: Adapting Multilingual Language Models to New Scripts Jonas Pfeiffer1, Ivan Vuli´c2, Iryna Gurevych1, Sebastian Ruder3 1Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt 2Language Technology Lab, University of Cambridge 3DeepMind pfeiffer@ukp.tu-darmstadt.de Abstract Massively multilingual language models such as multilingual BERT offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. However, due to limited ca- pacity and large differences in pretraining data sizes, there is a profound performance gap be- tween resource-rich and resource-poor target languages. The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts un- seen during pretraining. In this work, we pro- pose a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low- resource languages and unseen scripts. Rely- ing on matrix factorization, our methods cap- italize on the existing latent knowledge about multiple languages already available in the pre- trained model’s embedding matrix. Further- more, we show that learning of the new dedi- cated embedding matrix in the target language can be improved by leveraging a small num- ber of vocabulary items (i.e., the so-called lexically overlapping tokens) shared between mBERT’s and target language vocabulary. Our adaptation techniques offer substantial perfor- mance gains for languages with unseen scripts. We also demonstrate that they can yield im- provements for low-resource languages writ- ten in scripts covered by the pretrained model. 1 Introduction Massively multilingual language models pretrained on large multilingual data, such as multilingual BERT (mBERT; Devlin et al., 2019) and XLM-R (Conneau et al., 2020a) are the current state-of-the- art vehicle for effective cross-lingual transfer (Hu et al., 2020). However, while they exhibit strong transfer performance between resource-rich and similar languages (Conneau et al., 2020a; Artetxe et al., 2020), these models struggle with transfer to low-resource languages (Wu and Dredze, 2020) and languages not represented at all in their pre- Amharic (am)Tibetan (bo) Khmer (km) Divehi (dv) Sinhala (si) የእግዚአብሔርን ཉྩཡོནིརོདྷཨེཝ មានសេចក្តីត្រិះរិះރައީސުލްޖުމްހޫރިއްޔާގެ ප්රජාතන්ත්රවාදී የሚሰጡዋቸውን ྃབཱདཱིམཧཱཤྲམཎ ត្រូវអាបត្ដិទុក្កដދެނެވަޑައިގަންނަވައެވެ දෙපාර්තමේන්තුව የእግዚአብሔር བཱདཱིམཧཱཤྲམཎ តាំងឡើងដោយសមុដ្ឋានއޮސްޓިއޯއާތުރައިޓިސް පුරාවිද්යාත්මක ቤተክርስቲያን ཧཱུྃཕཊསྭཱཧཱ ពួកឆព្វគ្គិយភិក្ខុރައީސުލްޖުމްހޫރިއްޔާ ක්රියාකාරිත්වය ኢትዮጵያውያን ཧཱུྃཧཱུྃཕཊ ដែលមិនទាន់ជ្រះថ្លាލިބިވަޑައިގެންފައިވާ ක්රියාකාරීත්වය በእግዚአብሔር བྷྱཿཤཱནྟིཾ ភិក្ខុត្រូវអាបត្តិކަނާތްފަރާތުންނެވެ විශ්වවිද්යාලයේ እግዚአብሔርን བཻཌཱུརྱའི ឃុំមួយស្ថិតនៅក្នុងޖުމްހޫރިއްޔާއެކެވެ දිස්ත්රික්කයේ ከእግዚአብሔር ེདྷརྨཱཧེཏ របស់ព្រះមានព្រះភាគލިބިވަޑައިގެންނެވި සම්පූර්ණයෙන්ම የመጀመሪያውን ཝཱཧེཏུནྟེ ធ្វើឲ្យជាក់ច្បាស់ބަޔާންކޮށްފައިވަނީ පාර්ලිමේන්තුව ትምህርታቸውን སྤྲེའུའི ធ្វើទុកក្នុងចិត្އުފަންވެވަޑައިގަ විශ්වවිද්යාලය Amharic Tibetan Divehi የእግዚአብሔርን ཉྩཡོནིརོདྷཨེཝރައީސުލްޖުމްހޫރިއްޔާގެ የሚሰጡዋቸውን ྃབཱདཱིམཧཱཤྲམཎދެނެވަޑައިގަންނަވައެވެ የእግዚአብሔር བཱདཱིམཧཱཤྲམཎއޮސްޓިއޯއާތުރައިޓިސް ቤተክርስቲያን ཧཱུྃཕཊསྭཱཧཱރައީސުލްޖުމްހޫރިއްޔާ ኢትዮጵያውያን ཧཱུྃཧཱུྃཕཊލިބިވަޑައިގެންފައިވާ በእግዚአብሔር བྷྱཿཤཱནྟིཾކަނާތްފަރާތުންނެވެ እግዚአብሔርን བཻཌཱུརྱའིޖުމްހޫރިއްޔާއެކެވެ ከእግዚአብሔር ེདྷརྨཱཧེཏލިބިވަޑައިގެންނެވި የመጀመሪያውን ཝཱཧེཏུནྟེބަޔާންކޮށްފައިވަނީ ትምህርታቸውን སྤྲེའུའིއުފަންވެވަޑައިގަ Figure 1: Example tokens of unseen scripts. training corpora (Pfeiffer et al., 2020b; Müller et al., 2021; Ansell et al., 2021). The most extreme chal- lenge is dealing with unseen languages with unseen scripts (i.e., the scripts are not represented in the pretraining data; see Figure 1), where the pretrained models are bound to fail entirely if they are used off-the-shelf without any further model adaptation. Existing work focuses on the embedding layer and learns either a new embedding matrix for the target language (Artetxe et al., 2020) or adds …

Content truncated to 500 words