When the Student Becomes the Master: Learning Better and Smaller Monolingual Models from mBERT

Proceedings of the 29th International Conference on Computational Linguistics, pages 4434–4441 October 12–17, 2022. 4434 When the Student Becomes the Master: Learning Better and Smaller Monolingual Models from mBERT Pranaydeep Singh and Els Lefever LT3, Language and Translation Technology Team Department of Translation, Interpreting and Communication – Ghent University Groot-Brittanniëlaan 45, 9000 Ghent, Belgium {firstname.lastname}@ugent.be Abstract In this research, we present pilot experiments to distil monolingual models from a jointly trained model for 102 languages (mBERT). We demonstrate that it is possible for the target lan- guage to outperform the original model, even with a basic distillation setup. We evaluate our methodology for 6 languages with varying amounts of resources and belonging to different language families. 1 Introduction The introduction of the Transformer architecture, which aims to solve sequence-to-sequence tasks while also handling long-range dependencies re- lying on self-attention (Vaswani et al., 2017), has caused a huge improvement on the state-of-the- art for a wide range of natural language process- ing tasks. Despite the clear gain in performance, these pre-trained language models are extremely data- and computation-hungry and their sizes keep growing at an incredible speed. For example, RoBERTa (Liu et al., 2019) is trained on a text corpus of 160 GB, and the recent collaboration of Microsoft and Nvidia resulted in a language model containing 530 billion parameters that was trained on 270 billion tokens (Smith et al., 2022). Besides the ethical questions on sustainability raised by the NLP community, these huge language models also pose a lot of operational challenges, as they require massive amounts of training data, com- putation power and storage capacity. More im- portantly, the limits on training data also intro- duce limits on the languages these models can work with. Consequently, English has been the default language newer transformers have been trained on. More recently, much attention has been devoted to multilingual aspects, especially with the advent of joint models like mBERT (De- vlin et al., 2019) and XLM (Conneau and Lam- ple, 2019). However, the percentage of data used to train joint models for low-resourced languages when compared to English is orders of magni- tudes lesser and results in poorer representations for lower-resourced languages (Wu and Dredze, 2020). Moreover, when language representations for a specific low-resourced language are needed, there are no available monolingual models for a lot of these languages, and the entire large jointly- trained model needs to be loaded. More recently, a lot of attention is paid to sustainability for trans- formers, resulting in various approaches like prun- ing, weight sharing and distillation to reduce the size of large models. Previous research has shown good results for task-specific distillation, distilla- tion from larger English BERT models and dis- tillation from mBERT into a smaller multilingual model (see Section 2). In this paper, we attempt to combine these two research directions and distil smaller monolingual transformers from mBERT. Our hypothesis was that distilling would help to improve the representations for a target language, as the model can focus its prediction power …

Content truncated to 500 words