Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6776 - 6791 December 7-11, 2022 ©2022 Association for Computational Linguistics Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding Rishabh Bhardwaj∗ §‡† Amrita Saha∗ § Steven C.H. Hoi § Soujanya Poria ‡ §Salesforce Research ‡Singapore University of Technology and Design Abstract Prompt Tuning has been largely successful as a parameter-efficient method of conditioning large-scale pre-trained language models to per- form downstream tasks. Thus far, soft prompt tuning learns a fixed set of task-specific con- tinuous vectors, i.e., soft tokens that remain static across the task samples. A fixed prompt, however, may not generalize well to the diverse kinds of inputs the task comprises. In order to address this, we propose Vector-quantized Input-contextualized Prompts (VIP)1 as an ex- tension to the soft prompt tuning framework. VIP particularly focuses on two aspects— contextual prompts that learns input-specific contextualization of the soft prompt tokens through a small-scale sentence encoder and quantized prompts that maps the contextu- alized prompts to a set of learnable codebook vectors through a Vector quantization network. On various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI, VIP outperforms the soft prompt tun- ing (PT) baseline by an average margin of 1.19%. Further, our generalization studies show that VIP learns more robust prompt represen- tations, surpassing PT by a margin of 0.6% - 5.3% on Out-of-domain QA and NLI tasks re- spectively, and by 0.75% on Multi-Task setup over 4 tasks spanning across 12 domains. 1 Introduction With the increase in network size, it has become more computationally expensive to fine-tune pre- trained language models (PLMs) on downstream tasks. Recent studies have shown that prompt- based learning is a quite effective parameter- efficient method of conditioning the behavior of PLMs for a given predictive task (Lester et al., ∗Equal contribution. † This work was performed while the first author was interning at Salesforce Research. Correspondence to: <amrita.saha@salesforce.com> 1Our implementation of VIP is made publicly available https://github.com/declare-lab/VIP Figure 1: Comparison of model performance vs param- eter size on SuperGLUE dataset. PT and FT denotes prompt tuning (2021) and fine-tuning respectively while Adapter (2019) learns adapter modules inserted be- tween layers of the PLM. VIP-IDP denotes instance dependent prompt generation (2022) under text-to-text framework. VIP is our proposed approach of quantized contextual soft prompts. 2021; Wu et al., 2022). The goal of prompt-based learning, in contrast to traditional supervised learn- ing approaches, is to find the task-specific template that is augmented to the input text in order to guide the PLM toward generating the correct output (Liu et al., 2021). The initial efforts in this direction involved manually designed templates to solve nat- ural language processing (NLP) tasks (Brown et al., 2020; Schick and Schütze, 2021, 2020). In order to reduce the human effort required in finding appropriate task and LM-specific prompt templates, more recent works aim to automate the process of prompt designing. One of the popu- lar ways is to search for templates in …

Content truncated to 500 words