Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11100–11119 July 9-14, 2023 ©2023 Association for Computational Linguistics Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models Junmo Kang, Wei Xu, Alan Ritter Georgia Institute of Technology junmo.kang@gatech.edu {wei.xu, alan.ritter}@cc.gatech.edu Abstract Fine-tuning large models is highly effective, however, inference can be expensive and pro- duces carbon emissions. Knowledge distilla- tion has been shown to be a practical solution to reduce inference costs, but the distillation pro- cess itself requires significant computational resources. Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manu- ally label additional fine-tuning data. In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small). We further investigate how the optimal budget allocated towards computation varies across scenarios. We will make our code, datasets, annotation cost estimates, and base- line models available as a benchmark to support further work on cost-efficient training of com- pact models. 1 Introduction Increasing the size of pre-trained models can con- sistently improve performance on downstream tasks after fine-tuning, as seen in studies based on BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2019), T5 (Raffel et al., 2020), and the work on empirical scaling laws (Brown et al., 2020; Lester et al., 2021; Her- nandez et al., 2021). However, using large mod- els for inference is expensive and contributes to carbon emissions (Patterson et al., 2021). To ad- dress this, researchers have explored methods to compress large models through techniques such as knowledge distillation (Hinton et al., 2015; Sanh et al., 2019; Gou et al., 2021), which is effective in reducing inference costs (Magister et al., 2022) Large Model Small Model Training Efficient Inference (Fixed Model Size) Small Model (+Annotated Data) Distillation (Soft Label) Small Model (Distilled) vs. Direct Use Fixed Budget More Data More Compute (GPUs) Figure 1: An illustration of two practical strategies to build a compact fixed-size model. Given a fixed budget and a small amount of initially annotated data, (i) one can annotate more data to directly fine-tune a small model. (ii) Alternatively, one may leverage a larger model with more computational resources to distill its knowledge into a small model for efficient inference. and improving the generalization of smaller stu- dent models (Stanton et al., 2021). Nonetheless, the distillation process itself still requires signifi- cant computational, memory, and storage resources (Xia et al., 2022). In addition to compressing models, an alternative approach to improve performance without increas- ing inference costs is to simply label additional data for fine-tuning. Recent work has shown that a few hundred extra labels can sometimes lead to …

Content truncated to 500 words