Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5064–5082, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5064 Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference Jian-Guo Zhang1∗ Kazuma Hashimoto2† Wenhao Liu2 Chien-Sheng Wu2 Yao Wan3 Philip S. Yu1 Richard Socher2 Caiming Xiong2 1 University of Illinois at Chicago, Chicago, USA 2Salesforce Research, Palo Alto, USA 3Huazhong University of Science and Technology, Wuhan, China {jzhan51,psyu}@uic.edu, wanyao@hust.edu.cn {k.hashimoto,wu.jason,rsocher,cxiong}@salesforce.com Abstract Intent detection is one of the core components of goal-oriented dialog systems, and detecting out-of-scope (OOS) intents is also a practically important skill. Few-shot learning is attracting much attention to mitigate data scarcity, but OOS detection becomes even more challeng- ing. In this paper, we present a simple yet ef- fective approach, discriminative nearest neigh- bor classification with deep self-attention. Un- like softmax classifiers, we leverage BERT- style pairwise encoding to train a binary clas- sifier that estimates the best matched train- ing example for a user input. We propose to boost the discriminative ability by transfer- ring a natural language inference (NLI) model. Our extensive experiments on a large-scale multi-domain intent detection task show that our method achieves more stable and accurate in-domain and OOS detection accuracy than RoBERTa-based classifiers and embedding- based nearest neighbor approaches. More no- tably, the NLI transfer enables our 10-shot model to perform competitively with 50-shot or even full-shot classifiers, while we can keep the inference time constant by leveraging a faster embedding retrieval model. 1 Introduction Intent detection is one of the core components when building goal-oriented dialog systems. The goal is to achieve high intent classification accu- racy, and another important skill is to accurately detect unconstrained user intents that are out-of- scope (OOS) in a system (Larson et al., 2019). A practical challenge is data scarcity because differ- ent systems define different sets of intents, and thus few-shot learning is attracting much attention. However, previous work has mainly focused on the ∗Work done while the first author was an intern at Sales- force Research. †Corresponding author. Figure 1: tSNE visual comparison for OOS detection between existing methods ((a) and (b)) and our pro- posed method ((c) and (d)). Their embeddings before their classifier layers are used (best viewed in color). few-shot intent classification without OOS (Luo et al., 2018; Casanueva et al., 2020). OOS detection can be considered as out-of- distribution detection (Hendrycks and Gimpel, 2017; DeVries and Taylor, 2018). Recent work has shown that large-scale pre-trained models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) still struggle with out-of-distribution detection, despite their strong in-domain perfor- mance (Hendrycks et al., 2020). Figure 1 (a) shows how unseen input text is mapped into a feature space, by a RoBERTa-based model for 15-way 5-shot intent classification. The separation be- tween OOS and some in-domain intents is not clear, which presumably hinders the model’s OOS detec- tion ability. This observation calls for investigation into more sample-efficient approaches to handling 5065 the in-domain and OOS …

Content truncated to 500 words