Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1906–1912 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1906 Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning Jian-Guo Zhang1∗, Trung Bui2, Seunghyun Yoon2, Xiang Chen2, Zhiwei Liu1 Congying Xia1, Quan Hung Tran2, Walter Chang2, Philip Yu1 1 University of Illinois at Chicago, Chicago, USA 2Adobe Research, San Jose, USA {jzhan51, zliu213, cxia8, psyu}@uic.edu, {bui, syoon, xiangche, qtran, wachang}@adobe.com Abstract In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically sim- ilar. We present a simple yet effective few- shot intent detection schema via contrastive pre-training and fine-tuning. Specifically, we first conduct self-supervised contrastive pre- training on collected intent datasets, which implicitly learns to discriminate semantically similar utterances without using any labels. We then perform few-shot intent detection together with supervised contrastive learn- ing, which explicitly pulls utterances from the same intent closer and pushes utterances across different intents farther. Experimen- tal results show that our proposed method achieves state-of-the-art performance on three challenging intent detection datasets under 5- shot and 10-shot settings. 1 Introduction Intent detection, aiming to identify intents from user utterances, is a key component in task-oriented dialog systems. In real systems such as Amazon Alexa, correctly identifying user intents is crucial for downstream tasks (Zhang et al., 2020b; Ham et al., 2020). A practical challenge is data scarcity as it is expensive to annotate enough examples for emerging intents, and how to accurately identify intents in few-shot learning has raised attention. Existing methods address the few-shot intent de- tection tasks mainly from two perspectives: (1) data augmentation and (2) task-adaptive training with pre-trained models. For the first category, Zhang et al. (2020a) and Mehri et al. (2020b) pro- pose a nearest neighbor classification schema with full use of the limited training examples in both training and inference stages. Xia et al. (2020b) and Peng et al. (2020) propose to generate utter- ances for emerging intents based on variational ∗Work done while the first author was an intern at Adobe Research. autoencoder (Kingma and Welling, 2013) and GPT- 2 (Radford et al., 2019), respectively. For the sec- ond category, Casanueva et al. (2020) and Mehri et al. (2020a) conduct intent detection by lever- aging related conversational pre-training models based on a few hundred million conversations. Meanwhile, they devise a task-adaptive training schema where the model is pre-trained on all rela- tive intent datasets or the target intent datasets with mask language modeling. However, previous methods such as data aug- mentation related models (Liu et al., 2021c) are inefficient for training and hard to scale to tasks with lots of intents. Moreover, these models do not tackle well the following scenarios: In real scenarios, the few-shot intent detection could be more challenging when there exist many fine- grained intents, especially semantically similar in- tents. For instance, BANKING77 (Casanueva et al., 2020) has a single domain with 77 intents, and CLINC150 (Larson et al., 2019) has …

Content truncated to 500 words