The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1846–1857 May 2-6, 2023 ©2023 Association for Computational Linguistics The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification Mohsen Mesgar1∗ Thy Thy Tran1∗ Goran Glavaš2 Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt 2CAIDAS, University of Würzburg, Germany www.ukp.tu-darmstadt.de Abstract In task-oriented dialog (ToD) new intents emerge on regular basis, with a handful of avail- able utterances at best. This renders effective Few-Shot Intent Classification (FSIC) a central challenge for modular ToD systems. Recent FSIC methods appear to be similar: they use pretrained language models (PLMs) to encode utterances and predominantly resort to nearest- neighbor-based inference. However, they also differ in major components: they start from different PLMs, use different encoding archi- tectures and utterance similarity functions, and adopt different training regimes. Coupling of these vital components together with the lack of informative ablations prevents the identifi- cation of factors that drive the (reported) FSIC performance. We propose a unified framework to evaluate these components along the follow- ing key dimensions: (1) Encoding architec- tures: Cross-Encoder vs Bi-Encoders; (2) Simi- larity function: Parameterized (i.e., trainable) vs non-parameterized; (3) Training regimes: Episodic meta-learning vs conventional (i.e., non-episodic) training. Our experimental re- sults on seven FSIC benchmarks reveal three new important findings. First, the unexplored combination of cross-encoder architecture and episodic meta-learning consistently yields the best FSIC performance. Second, episodic train- ing substantially outperforms its non-episodic counterpart. Finally, we show that splitting episodes into support and query sets has a lim- ited and inconsistent effect on performance. Our findings show the importance of ablations and fair comparisons in FSIC. We publicly re- lease our code and data1. 1 Introduction Intent classification deals with assigning one label from a predefined set of classes or intents to user ∗ Equal contribution. 1https://github.com/UKPLab/ eacl2023-few-shot-intent-classification utterances. This task is vital for task-oriented dia- log (ToD) systems since the predicted intent of an utterance is an essential input to other modules (i.e., dialog management) in these systems (Ma et al., 2022; Louvan and Magnini, 2020; Razumovskaia et al., 2021). Although intent classification has been widely studied, it still represents a challenge in settings where dialogue systems, including their intent classifiers, need to have the ability to be quickly adjusted to new domains and intent classes. The main challenges in training intent classifiers in such settings lies in the costly labeling of ut- terances (Zhang et al., 2022a; Wen et al., 2017; Budzianowski et al., 2018; Rastogi et al., 2020; Hung et al., 2022; Mueller et al., 2022). Few-shot intent classification (FSIC), which deals with ad- justing intent classifiers to new intents given only a handful of labeled instances, is thus of paramount importance for ToD systems. Various methods (§2) for FSIC have been pro- posed (Larson et al., 2019a; Casanueva et al., 2020a; Zhang et al., 2020; Mehri et al., 2020; Krone …

Content truncated to 500 words