Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1517 - 1529 December 7-11, 2022 ©2022 Association for Computational Linguistics Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery Yutao Mou1∗, Keqing He2∗, Pei Wang1, Yanan Wu1 Jingang Wang2, Wei Wu2, Weiran Xu1∗ 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan, Beijing, China {myt,wangpei,yanan.wu,xuweiran}@bupt.edu.cn {hekeqing,wangjingang,wuwei}@meituan.com Abstract Discovering out-of-domain (OOD) intent is important for developing new skills in task- oriented dialogue systems. The key chal- lenges lie in how to transfer prior in-domain (IND) knowledge to OOD clustering, as well as jointly learn OOD representations and cluster assignments. Previous methods suffer from in- domain overfitting problem, and there is a nat- ural gap between representation learning and clustering objectives. In this paper, we pro- pose a unified K-nearest neighbor contrastive learning framework to discover OOD intents. Specifically, for IND pre-training stage, we pro- pose a KCL objective to learn inter-class dis- criminative features, while maintaining intra- class diversity, which alleviates the in-domain overfitting problem. For OOD clustering stage, we propose a KCC method to form compact clusters by mining true hard negative sam- ples, which bridges the gap between clustering and representation learning. Extensive experi- ments on three benchmark datasets show that our method achieves substantial improvements over the state-of-the-art methods. 1 1 Introduction Out-of-domain (OOD) intent discovery aims to group new unknown intents into different clusters, which helps identify potential development direc- tions and develop new skills in a task-oriented dia- logue system (Lin et al., 2020; Zhang et al., 2021; Mou et al., 2022). Different from traditional text clustering task, OOD discovery considers how to leverage the prior knowledge of known in-domain (IND) intents to enhance discovering unknown OOD intents, which makes it challenging to di- rectly apply existing clustering algorithms (Mac- Queen, 1967; Xie et al., 2016; Chang et al., 2017; Caron et al., 2018) to the OOD discovery task. ∗The first two authors contribute equally. Weiran Xu is the corresponding author. 1We release our code at https://github.com/myt517/ KCOD a) KNN-based Contrastive Pre-training In-domain samples SCL+CE KCL+ CE A B A1 A2 B1 B2 Data Augmentation Pull Together b) KNN Contrastive Clustering (Cluster 1) (Cluster 1) C C1 C2 (Cluster 2) D D1 D2 (Cluster 3) (Other clusters) Push Away Pull Together Push Away Threshold KCC Instance-level CL Figure 1: The high-level idea of our KCOD compared to baselines. Fig (a) shows KCL increases the intra-class variance by only taking the K nearest samples of the same class as positive samples. Fig (b) displays that KCC filters out false negative samples that belong to the same class as the anchor and selects K nearest neighbors as true hard negatives to form clear cluster boundaries. The key challenges of OOD intent discovery come from two aspects: (1) Knowledge Trans- ferability. It requires transferring in-domain prior knowledge to help downstream OOD clus- tering. Early unsupervised intent discovery meth- ods (Hakkani-Tür et al., 2015; Padmasundari and Bangalore, 2018; Shi et al., …

Content truncated to 500 words