Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7544 - 7559 December 7-11, 2022 ©2022 Association for Computational Linguistics Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders Ivan Vuli´c, Iñigo Casanueva, Georgios Spithourakis, Avishek Mondal, Tsung-Hsien Wen and Paweł Budzianowski PolyAI Limited London, United Kingdom poly.ai Abstract Deploying task-oriented dialog (TOD) sys- tems for new domains and tasks requires nat- ural language understanding models that are 1) resource-efficient and work under low-data regimes; 2) adaptable, efficient, and quick- to-train; 3) expressive and can handle com- plex TOD scenarios with multiple user in- tents in a single utterance. Motivated by these requirements, we introduce a novel framework for multi-label intent detection (mID): MULTI-CONVFIT (Multi-Label Intent Detection via Contrastive Conversational Fine- Tuning). While previous work on efficient single-label intent detection learns a classi- fier on top of a fixed sentence encoder (SE), we propose to 1) transform general-purpose SEs into task-specialized SEs via contrastive fine-tuning on annotated multi-label data, 2) where task specialization knowledge can be stored into lightweight adapter modules with- out updating the original parameters of the in- put SE, and then 3) we build improved mID classifiers stacked on top of fixed specialized SEs. Our main results indicate that MULTI- CONVFIT yields effective mID models, with large gains over non-specialized SEs reported across a spectrum of different mID datasets, both in low-data and high-data regimes. 1 Introduction Task-oriented dialog (TOD) systems allow users to interact with computer applications through con- versation in order to solve a particular task with well-defined semantics (Levin and Pieraccini, 1995; Young, 2010). TOD supports a multitude of appli- cations such as automating different customer ser- vice tasks, facilitating bookings in hospitality and travel industries, or providing assistance in health- care or finance (Raux et al., 2003; El Asri et al., 2017; Xu et al., 2017; Budzianowski et al., 2018). Intent detection (ID), a task of recognizing the user’s intent or goal from their utterance, is a cru- cial component of any TOD system (Hemphill et al., 1990; Tür et al., 2010; Coucke et al., 2018). Intent detectors that adhere to industry standards should satisfy the following three requirements (Casanueva et al., 2020; Larson and Leach, 2022).1 (R1) Scalability and resource efficiency. They must be quickly bootstrapped for new domains and tasks. However, this process requires creating expensive annotations for each domain and task of interest, which calls for sample-efficient ID methods that achieve strong performance in low-data scenarios. (R2) Lightweight design and modularity. While large laguage models (LMs) have shown strong performance in the ID task, running full-fledged fine-tuning and storing separate fine-tuned models per each domain or task yields prohibitive storage and memory costs (Ding et al., 2022). Enabling fast training of intent detectors (Casanueva et al., 2020) also speeds up TOD development cycles. (R3) Expressiveness: supporting complex TOD sce- narios. Previous work has typically focused on more limited single-label ID scenarios (Liu et al., 2019a; Larson et al., 2019; Wu et al., 2020; Mehri et …

Content truncated to 500 words