LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency

Proceedings of the 29th International Conference on Computational Linguistics, pages 3120–3126 October 12–17, 2022. 3120 LayerConnect: Hypernetwork-Assisted Inter-Layer Connector to Enhance Parameter Efficiency Haoxiang Shi1∗†, Rongsheng Zhang2*, Jiaan Wang3†, Cen Wang4 Yinhe Zheng5 and Tetsuya Sakai1 1 Waseda University, Tokyo, Japan 2 Fuxi AI Lab, NetEase Inc., Hangzhou, China 3 Soochow University, Suzhou, China 4 KDDI Research Inc., Japan 5 Lingxin AI, China hollis.shi@toki.waseda.jp, tetsuya@waseda.jp, zhangrongsheng@corp.netease.com jawang1@stu.suda.edu.cn, ce-wang@kddi-research.jp, zhengyinhe1@163.com Abstract Pre-trained Language Models (PLMs) are the cornerstone of the modern Natural Language Processing (NLP). However, as PLMs become heavier, fine tuning all their parameters loses their efficiency. In this paper, we propose Lay- erConnect (hypernetwork-assisted inter-layer connectors) to enhance inference efficiency. Specifically, a light-weight connector with a linear structure is inserted between two Trans- former layers, and the parameters inside each connector are tuned by a hypernetwork com- prising an interpolator and a down-sampler. We conduct extensive experiments on the widely used GLUE benchmark. The experimental results verify the inference efficiency of our model. Compared to Adapter, with our struc- ture, parameters are reduced to approximately 11.75% for base PLMs and 8.82% for large PLMs, while the performance degradation is kept to less than 5% (2.07 points on average). 1 Introduction The emergence of pre-trained language models (PLMs) has brought Natural Language Process- ing (NLP) to a new era (Qiu et al., 2020). Fine tuning all PLMs parameters has become the most common strategy to apply them to downstream tasks (Zheng et al., 2021a; Lai et al., 2021; Wang et al., 2021, 2022; Chen et al., 2022; He et al., 2022; Zheng et al., 2020; Wang et al., 2020; Zheng et al., 2021b,c; Zhang et al., 2020; Zhou et al., 2021). However, in this manner, one should store a full copy of a PLM for each downstream task because all its parameters are updated during the fine-tuning stage. Thus, this strategy challenges the usage of the PLMs on edge devices (e.g., mobile phone and embedded systems) that have limited storage space. Under this background, parameter efficiency has gradually attracted attention from both the research ∗These two authors contributed equally. † This work is mainly done when SHI and WANG were interning at NetEase Inc.. community and industries (Houlsby et al., 2019; Li and Liang, 2021; Zaken et al., 2021; Liu et al., 2021; Ding et al., 2022; Zhang et al., 2021). The core concept of parameter efficiency is to reduce the trainable parameters of PLMs. Consequently, when deploying a PLM, only the different trained parameters from multiple tasks must be stored, to- gether with one copy of the shared frozen parame- ters. Currently, there are three main types of param- eter efficiency methods: (1) Specification-based methods train only a small part of the parameters in the original PLM, while the others are frozen. For example, Bitfit (Zaken et al., 2021) only updates the parameters of the bias and task-specific layers in PLMs. (2) Prompt-based methods prepend addi- tional context (i.e., prompts) to the original input, and change only the prompt …

Content truncated to 500 words