MultiFiT: Efficient Multi-lingual Language Model Fine-tuning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5702–5707, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5702 MultiFiT: Efficient Multi-lingual Language Model Fine-tuning Julian Eisenschlos1†∗ Sebastian Ruder2,3‡∗ Piotr Czapla4∗ Marcin Kardas4∗ Sylvain Gugger5 Jeremy Howard5,6 1ASAPP, Inc. 2National University of Ireland 3Aylien Ltd., Dublin 4n-waves, Wrocław 5fast.ai 6University of San Francisco Abstract Pretrained language models are promising par- ticularly for low-resource languages as they only require unlabelled data. However, train- ing existing models requires huge amounts of compute, while pretrained cross-lingual mod- els often underperform on low-resource lan- guages. We propose Multi-lingual language model Fine-Tuning (MultiFiT) to enable prac- titioners to train and fine-tune language mod- els efficiently in their own language. In ad- dition, we propose a zero-shot method us- ing an existing pretrained cross-lingual model. We evaluate our methods on two widely used cross-lingual classification datasets where they outperform models pretrained on orders of magnitude more data and compute. We release all models and code1. 1 Introduction Pretrained language models (LMs) have shown striking improvements on a range of natural lan- guage processing (NLP) tasks (Peters et al., 2018a; Howard and Ruder, 2018; Devlin et al., 2018). These models only require unlabelled data for training and are thus particularly useful in scenar- ios where labelled data is scarce. As much of NLP research has focused on the English language, the larger promise of these models is to bridge the dig- ital language divide2 and enable the application of NLP methods to many of the world’s other 6,000 languages where labelled data is less plentiful. Recently, cross-lingual extensions of these LMs have been proposed that train on multiple lan- guages jointly (Artetxe and Schwenk, 2018; Lam- ple and Conneau, 2019). These models are able 1http://nlp.fast.ai ⋆The first four authors contributed equally. †Corresponding author: eisenjulian@gmail.com ‡Sebastian is now affiliated with DeepMind. 2http://labs.theguardian.com/ digital-language-divide/ to perform zero-shot learning, only requiring la- belled data in the source language. However, source data in another language may often not be available, whereas obtaining a small number of la- bels is typically straightforward. Furthermore such models have several down- sides: a) some variants rely on large amounts of parallel data, which may not be available for truly low-resource languages; b) they require a huge amount of compute for training3; and c) cross-lingual models underperform on low- resource languages—precisely the setting where they would be most useful. We are aware of two possible reasons for this: 1) Languages that are less frequently seen during training are un- derrepresented in the embedding space.4 2) In- frequent scripts are over-segmented in the shared word piece vocabulary (Wang et al., 2019). In this work, we show that small monolin- gual LMs are able to outperform expensive cross- lingual models both in the zero-shot and the super- vised setting. We propose Multi-lingual language model Fine-tuning (MultiFit) to enable practition- ers to train and fine-tune language models effi- ciently.5 Our model combines …

Content truncated to 500 words