MergeDistill: Merging Language Models using Pre-trained Distillation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2874–2887 August 1–6, 2021. ©2021 Association for Computational Linguistics 2874 MERGEDISTILL: Merging Pre-trained Language Models using Distillation Simran Khanuja Google Research India simrankh@google.com Melvin Johnson Google Research USA melvinp@google.com Partha Talukdar Google Research India partha@google.com Abstract Pre-trained multilingual language models (LMs) have achieved state-of-the-art results in cross-lingual transfer, but they often lead to an inequitable representation of languages due to limited capacity, skewed pre-training data, and sub-optimal vocabularies. This has prompted the creation of an ever-growing pre- trained model universe, where each model is trained on large amounts of language or do- main specific data with a carefully curated, lin- guistically informed vocabulary. However, do- ing so brings us back full circle and prevents one from leveraging the benefits of multilin- guality. To address the gaps at both ends of the spectrum, we propose MERGEDISTILL, a framework to merge pre-trained LMs in a way that can best leverage their assets with mini- mal dependencies, using task-agnostic knowl- edge distillation. We demonstrate the applica- bility of our framework in a practical setting by leveraging pre-existing teacher LMs and train- ing student LMs that perform competitively with or even outperform teacher LMs trained on several orders of magnitude more data and with a fixed model capacity. We also highlight the importance of teacher selection and its im- pact on student model performance. 1 Introduction While current state-of-the-art multilingual lan- guage models (LMs) (Devlin et al., 2019; Conneau et al., 2020) aim to represent 100+ languages in a single model, efforts towards building monolin- gual (Martin et al., 2019; Kuratov and Arkhipov, 2019) or language-family based (Khanuja et al., 2021) models are only increasing with time (Rust et al., 2020). A single model is often incapable of effectively representing a diverse set of languages, evidence of which has been provided by works highlighting the importance of vocabulary curation and size (Chung et al., 2020; Artetxe et al., 2020), Figure 1: Previous works (left) typically focus on combining fine-tuned models derived from a single pre-trained model using distillation. We propose MERGEDISTILL to combine pre-trained teacher LMs from multiple monolingual/multilingual LMs into a sin- gle multilingual task-agnostic student LM. pre-training data volume (Liu et al., 2019a; Con- neau et al., 2020), and the curse of multilinguality (Conneau et al., 2020). Language specific mod- els alleviate these issues with a custom vocabulary which captures language subtleties1 and large mag- nitudes of pre-training data scraped from several domains (Virtanen et al., 2019; Antoun et al., 2020). However, building language specific LMs brings us back to where we started, preventing us from leveraging the benefits of multilinguality like zero- shot task transfer (Hu et al., 2020), positive trans- fer between related languages (Pires et al., 2019; Lauscher et al., 2020) and an ability to handle code- mixed text (Pires et al., 2019; Tsai et al., 2019). We need an approach that encompasses the best of both worlds, i.e., leverage the capabilities of the powerful language-specific LMs while still …

Content truncated to 500 words