Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks

Proceedings of the 29th International Conference on Computational Linguistics, pages 7053–7064 October 12–17, 2022. 7053 Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks Xin Zhou⋆∗, Ruotian Ma⋆∗, Yicheng Zou⋆, Xuanting Chen⋆, Tao Gui♦†, Qi Zhang⋆♣†, Xuanjing Huang⋆, Rui Xie♠, Wei Wu♠ ⋆School of Computer Science, Fudan University ♦Institute of Modern Languages and Linguistics, Fudan University ♣Shanghai Key Laboratory of Intelligent Information Processing ♠Meituan Inc., Beijing, China {xzhou20,rtma19,tgui,qz,xjhuang}@fudan.edu.cn Abstract Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. Recent studies have ex- plored parameter-efficient PLM tuning, which only updates a small amount of task-specific parameters while achieving both high efficiency and comparable performance against standard fine-tuning. However, all these methods ignore the inefficiency problem caused by the task- specific output layers, which is inflexible for us to re-use PLMs and introduces non-negligible parameters. In this work, we focus on the text classification task and propose plugin-tuning, a framework that further improves the efficiency of existing parameter-efficient methods with a unified classifier. Specifically, we re-formulate both token and sentence classification tasks into a unified language modeling task, and map label spaces of different tasks into the same vocabulary space. In this way, we can directly re-use the language modeling heads of PLMs, avoiding introducing extra parameters for different tasks. We conduct experiments on six classification benchmarks. The exper- imental results show that plugin-tuning can achieve comparable performance against fine- tuned PLMs, while further saving around 50% parameters on top of other parameter-efficient methods. 1 Introduction In many industrial applications about natural lan- guage processing, it becomes a de-facto paradigm that we first pre-train large-scale language models (PLM) (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2019) on external corpora and then fine-tune them on target tasks. However, each fine- tuned PLM is generally applicable to only one task. As the number of applications increases, deploying independent instances of fine-tuned PLMs for different tasks significantly increases the ∗ Equal Contribution. † Corresponding authors. Frozen PLM SA Plugin < TEXT > NER Plugin < TEXT > NLI Plugin < TEXT > POS Plugin < TEXT > SA OUTPUT NER OUTPUT NLI OUTPUT POS OUTPUT Use Plugin to Perform Task Figure 1: An intuitive overview of plugin-tuning. Plugin tuning only trains and stores a parameter-efficient plugin rather than a full PLM for each task. Different tasks share a unified output layer despite different label spaces. computation and storage costs. For instance, GPT- 3 (Brown et al., 2020) contains 175B parameters, which makes it almost impossible to fine-tune and deploy GPT-3 for target tasks. Even if large-scale fine-tuned PLMs are available, they are also not conducive to community distribution and sharing. Recently, a new branch of researches named parameter-efficient tuning (Guo et al., 2021; Zaken et al., 2021; Hu et al., 2022) have received much attention in the NLP community. Compared to standard fine-tuning, these methods fine-tune only a small portion of the model parameters while keeping most of the PLM parameters frozen (Ding et al., 2022). …

Content truncated to 500 words