Label Semantic Aware Pre-training for Few-shot Text Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8318 - 8334 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Label Semantic Aware Pre-training for Few-shot Text Classification Aaron Mueller1∗ Jason Krone2 Salvatore Romeo2 Saab Mansour2 Elman Mansimov2 Yi Zhang2 Dan Roth3,2 1Department of Computer Science, Johns Hopkins University 2Amazon Web Services AI Labs 3Department of Computer & Information Science, University of Pennsylvania amueller@jhu.edu, {kronej,romeos,saabm,mansimov,yizhngn,drot}@amazon.com Abstract In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by per- forming secondary pre-training on labeled sen- tences from a variety of domains. As domain- general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform exper- iments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! An- swers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings. 1 Introduction Large pre-trained language models have enabled better performance on many NLP tasks—especially in few-shot settings (Brown et al., 2020; Schick and Sch¨utze, 2021a; Wu and Dredze, 2020). More infor- mative representations of textual inputs often leads to much higher downstream performance on NLP applications, which explains the rapid and general adoption of models such as (Ro)BERT(a) (Devlin et al., 2019; Liu et al., 2019), GPT-2 (Radford et al., 2019), and T5 (Raffel et al., 2020). However, while these models are often used to effectively encode inputs, fewer works have attempted to give models access to informative representations of labels as well. ∗Work done as an intern at Amazon Web Services. Figure 1: Overview of our approach, label semantic aware pre-training (LSAP). We collect utterance-intent pairs and create new pairs from unlabeled Reddit and Twitter data, convert the intents to natural language, concatenate the utterance and intent, noise the con- catenated sequence, and train a sequence-to-sequence model to denoise the sequence. Most discriminative approaches to text classi- fication only give the model access to label in- dices. A recent stream of work has obtained signifi- cant improvements in structured prediction tasks by using sequence-to-sequence (seq2seq) models to generate labels (Athiwaratkun et al., 2020; Paolini et al., 2021). Yet these generative approaches make use of label semantics—the meaning of class la- bel names—only during fine-tuning and predic- tion. Thus, we propose Label Semantic Aware Pre- training (LSAP) to incorporate label semantics as well as input-label associations into the pre-training step (Figure 1). Our experiments show that LSAP yields higher performance with fewer fine-tuning examples in a variety of domains. Our contributions include the following: 1. A …

Content truncated to 500 words