Distillation of encoder-decoder transformers for sequence labelling

Findings of the Association for Computational Linguistics: EACL 2023, pages 2539–2549 May 2-6, 2023 ©2023 Association for Computational Linguistics Distillation of encoder-decoder transformers for sequence labelling Marco Farina∗1 Duccio Pappadopulo∗1 Anant Gupta1 Leslie Huang1 Ozan ˙Irsoy1 Thamar Solorio†1,2 1Bloomberg 2Department of Computer Science, University of Houston {mfarina19, dpappadopulo, agupta968, lhuang328, oirsoy}@bloomberg.net tsolorio@uh.edu Abstract Driven by encouraging results on a wide range of tasks, the field of NLP is experiencing an ac- celerated race to develop bigger language mod- els. This race for bigger models has also under- scored the need to continue the pursuit of prac- tical distillation approaches that can leverage the knowledge acquired by these big models in a compute-efficient manner. Having this goal in mind, we build on recent work to propose a hallucination-free framework for sequence tagging that is especially suited for distillation. We show empirical results of new state-of-the- art performance across multiple sequence la- belling datasets and validate the usefulness of this framework for distilling a large model in a few-shot learning scenario. 1 Introduction Sequence labelling (SL) can be defined as the task of assigning a label to a span in the input text. Some examples of SL tasks are: i) named entity recog- nition (NER), where these labelled spans refer to people, places, or organizations, and ii) slot-filling, where these spans or slots of interest refer to at- tributes relevant to complete a user command, such as song name and playlist in a dialogue system. In general, these spans vary semantically depending on the domain of the task. Despite the strong trend in NLP to explore the use of large language models (LLMs) there is still limited work evaluating prompting and decoding mechanisms for SL tasks. In this paper we propose and evaluate a new inference approach for SL that addresses two practical constraints: • Data scarcity: The lack of vast amounts of annotated, and sometimes even the lack of unlabelled data, in the domain/language of interest. ∗Equal contribution †Research completed during sabbatical at Bloomberg • Restricted computing resources at infer- ence time: LLMs are very effective, but deploying them to production-level environ- ments is expensive, especially in contexts with latency constraints, such as in a live dialogue system. Data scarcity leads us to consider high- performing encoder-decoder based LLMs. We ad- dress deployment concerns by considering distil- lation of such models into much smaller SL archi- tectures, for instance Bi-Directional Long Short Memory (BiLSTM) (Hochreiter and Schmidhuber, 1997) units, through the use of both labelled and unlabelled data. A standard distillation approach, knowledge dis- tillation (KD) (Hinton et al., 2015), requires access to the probability that the teacher network assigns to each of the possible output tags. This proba- bility distribution is typically unavailable at infer- ence time for LLMs; thus, distillation of encoder- decoder models needs to resort to pseudo-labels:1 the student is trained on the one-hot labels that the teacher assigns to examples in an unlabelled dataset. This prevents the student model from learn- ing those relationships among the …

Content truncated to 500 words