MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1161–1173 May 2-6, 2023 ©2023 Association for Computational Linguistics MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers Mohammadmahdi Nouriborji2†, Omid Rohanian1,2†, Samaneh Kouchaki3, David A. Clifton1,4 1Department of Engineering Science, University of Oxford, Oxford, UK 2NLPie Research, Oxford, UK 3Dept. Electrical and Electronic Engineering, University of Surrey, Guildford, UK 4Oxford-Suzhou Centre for Advanced Research, Suzhou, China {m.nouriborji,omid}@nlpie.com samaneh.kouchaki@surrey.ac.uk {omid.rohanian,david.clifton}@eng.ox.ac.uk Abstract Pre-trained Language Models (LMs) have become an integral part of Natural Language Processing (NLP) in recent years, due to their superior performance in downstream applications. In spite of this resounding success, the usability of LMs is constrained by computational and time complexity, along with their increasing size; an issue that has been referred to as ‘overparameterisation’. Different strategies have been proposed in the literature to alleviate these problems, with the aim to create effective compact models that nearly match the performance of their bloated counterparts with negligible performance losses. One of the most popular techniques in this area of research is model distillation. Another potent but underutilised technique is cross-layer parameter sharing. In this work, we combine these two strategies and present MiniALBERT, a technique for converting the knowledge of fully parameterised LMs (such as BERT) into a compact recursive student. In addition, we investigate the application of bottleneck adapters for layer-wise adaptation of our recursive student, and also explore the efficacy of adapter tuning for fine-tuning of compact models. We test our proposed models on a number of general and biomedical NLP tasks to demonstrate their viability and com- pare them with the state-of-the-art and other existing compact models. All the codes used in the experiments are available at https: //github.com/nlpie-research/ MiniALBERT. Our pre-trained com- pact models can be accessed from https://huggingface.co/nlpie. †The two authors contributed equally to this work. 1 Introduction Following the introduction of BERT (Devlin et al., 2019), generic pre-trained Language Models (LMs) have started to dominate the field of NLP. Virtu- ally all state-of-the-art NLP models are built on top of some large pre-trained transformer as a back- bone and are subsequently fine-tuned on their target dataset. While this pre-train and fine-tune approach has resulted in significant improvements across a wide range of NLP tasks, the widespread use of resource-exhaustive and overparameterised trans- formers has also raised concerns among researchers about their energy consumption, environmental im- pact, and ethical implications (Strubell et al., 2019; Bender et al., 2021). As a response to this, different approaches have appeared with the aim to make large LMs more efficient, accessible, and environmentally friendly. Model compression is a line of research that has re- cently received considerable attention. It involves encoding a larger and slower but more performant model into a smaller and faster one with the aim to retain much of the former’s performance capability (Bucilua et al., 2006). Knowledge distillation (Hin- ton et al., 2015), quantisation (Shen et al., 2020), and pruning (Ganesh et al., 2021) are three exam- ples of such …

Content truncated to 500 words