Noisy Channel Language Model Prompting for Few-Shot Text Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5316 - 5330 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Noisy Channel Language Model Prompting for Few-Shot Text Classification Sewon Min1,2 , Mike Lewis,2 Hannaneh Hajishirzi1,3 , Luke Zettlemoyer1,2 1University of Washington 2Facebook AI Research 3Allen Institute for AI {sewon,hannaneh,lsz}@cs.washington.edu mikelewis@fb.com Abstract We introduce a noisy channel approach for lan- guage model prompting in few-shot text classi- fication. Instead of computing the likelihood of the label given the input (referred as di- rect models), channel models compute the con- ditional probability of the input given the la- bel, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning meth- ods with no or very limited updates to the lan- guage model parameters, via either in-context demonstration or prompt tuning. Our exper- iments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their sta- bility, i.e., lower variance and higher worst- case accuracy. We also present extensive abla- tions that provide recommendations for when to use channel prompt tuning instead of other competitive methods (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or general- ization to unseen labels is required. 1 Introduction Prompting large language models, by prepending natural language text or continuous vectors (called prompts) to the input, has shown to be promising in few-shot learning (Brown et al., 2020). Prior work has proposed methods for finding better prompt (Shin et al., 2020; Li and Liang, 2021; Lester et al., 2021) or better scoring of the output from the model (Zhao et al., 2021; Holtzman et al., 2021). These studies directly predict target tokens to deter- mine the prediction for an end task. Despite promis- ing results, they can be unstable with high variance across different verbalizers (text expression for la- bels) and seeds, and the worst-case performance is often close to random (Perez et al., 2021; Lu et al., 2021). In this paper, we introduce alternative channel models for prompted few-shot text classification LM =(“A three-hour cinema master class.”, “It was great.”) (x, y) A three-hour cinema master class. It was great. It was great. A three-hour cinema master class. P(y|x) Direct P(x|y)P(y) ∝ P(x|y) Channel Output Input Figure 1: An illustration of the direct model and the channel model for language model prompting in the sentiment analysis task. with large language models, inspired by noisy chan- nel models in machine translation (Brown et al., 1993; Koehn et al., 2003; Yu et al., 2017; Yee et al., 2019) and their extensions to other tasks (Yogatama et al., 2017; Lewis and Fan, 2018). Unlike direct models that compute the conditional probability of the label token given the input, channel models compute the conditional probability of the input given the output (Figure 1). Intuitively, channel models are required to explain …

Content truncated to 500 words