Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2580–2591, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2580 Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages Michael A. Hedderich1, David I. Adelani1, Dawei Zhu1, Jesujoba Alabi1,2, Udia Markus3 & Dietrich Klakow1 1Saarland University, Saarland Informatics Campus, Germany 2DFKI GmBH, Saarbr¨ucken, Germany 3Nuhu Bamalli Polytechnic, Zaira, Nigeria {mhedderich,didelani,dzhu,dietrich.klakow}@lsv.uni-saarland.de jesujoba oluwadara.alabi@dfki.de Abstract Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great im- provements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource lan- guages could not be easily transferred to re- alistic, low-resource scenarios. In this work, we study trends in performance for differ- ent amounts of available resources for the three African languages Hausa, isiXhosa and Yor`ub´a on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 la- beled sentences the same performance as base- lines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional exper- iments on assumptions such as time and hard- ware restrictions highlight challenges and op- portunities in low-resource learning. 1 Introduction Deep learning techniques, including contextualized word embeddings based on transformers and pre- trained on language modelling, have resulted in considerable improvements for many NLP tasks. However, they often require large amounts of la- beled training data, and there is also growing evidence that transferring approaches from high to low-resource settings is not straightforward. In (Loubser and Puttkammer, 2020a), rule-based or linguistically motivated CRFs still outperform RNN-based methods on several tasks for South African languages. For pretraining approaches where labeled data exists in a high-resource lan- guage, and the information is transferred to a low- resource language, Hu et al. (2020) find a signif- icant gap between performance on English and the cross-lingually transferred models. In a recent study, Lauscher et al. (2020) find that the trans- fer for multilingual transformer models is less ef- fective for resource-lean settings and distant lan- guages. A popular technique to obtain labeled data quickly and cheaply is distant and weak supervi- sion. Kann et al. (2020) recently inspected POS classifiers trained on weak supervision. They found that in contrast to scenarios with simulated low- resource settings of high-resource languages, in truly low-resource settings this is still a difficult problem. These findings also highlight the im- portance of aiming for realistic experiments when studying low-resource scenarios. In this work, we analyse multilingual trans- former models, namely mBERT (Devlin et al., 2019; Devlin, 2019) and XLM-RoBERTa (Con- neau et al., 2019). We evaluate both sequence and token classification tasks in the form of news ti- tle topic classification and named entity recogni- tion (NER). A variety of approaches have been proposed to improve performance in low-resource settings. In this work, we study (i) transfer learn- ing from a high-resource language and …

Content truncated to 500 words