Few-shot Pseudo-Labeling for Intent Detection

Proceedings of the 28th International Conference on Computational Linguistics, pages 4993–5003 Barcelona, Spain (Online), December 8-13, 2020 4993 Few-shot Pseudo-Labeling for Intent Detection Thomas Dopierre12 and Christophe Gravier1 and Julien Subercaze1 and Wilfried Logerais2 1Laboratoire Hubert Curien 2Meetic Saint-´Etienne, France Paris, France firstname.lastname@ univ-st-etienne.fr {t.dopierre,w.logerais}@ meetic-corp.com Abstract In this paper, we introduce a state-of-the-art pseudo-labeling technique for few-shot intent de- tection. We devise a folding/unfolding hierarchical clustering algorithm which assigns weighted pseudo-labels to unlabeled user utterances. We show that our two-step method yields significant improvement over existing solutions. This performance is achieved on multiple intent detec- tion datasets, even in more challenging situations where the number of classes is large or when the dataset is highly imbalanced. Moreover, we confirm these results on the more general text classification task. We also demonstrate that our approach nicely complements existing solu- tions, thereby providing an even stronger state-of-the-art ensemble method. We make our code publicly available1 for future research. 1 Introduction Labeling user utterances as intents is of paramount importance for chatbots. Intent classes are usually engineered by a linguistic team, and these classes are likely to evolve with time. The most frequent situations include new unanticipated intents found by dialogue traces analysis, the need to introduce new intents that were previously handled using web forms, or the urge to split existing intents into several new ones for fine-grained user interactions. Regardless the rationale, each time the intent referential evolves, a significant amount of new labeled data is required to train a decent intent detection model. This calls for a time-consuming and error-prone labeling process – it is therefore expensive. Consequently, intent detection systems need robust models in few-shot settings in order to avoid to repeatedly suffer from labeling user utterances into intents. A solution to this problem is to generate pseudo-labels for unlabeled utterances. A pseudo-label is a label (intent class) automatically assigned to an unlabeled utterance using the knowledge found in the set of labeled data. The resulting dataset composed of both labeled and pseudo-labeled utterances is then used to train a supervised intent detection model. Most pseudo-labeling methods combine advances in word representations (Peters et al., 2018; Bojanowski et al., 2017; Tissier et al., 2017) and then consider unsupervised clustering algorithms (von Luxburg, 2007) in order to propagate known labels to unlabeled data (see Section 2). Nonetheless, there are major drawbacks to existing pseudo-labeling algorithms. First, since clustering algorithms are used to partition user utterances representations, it leads to provide a pseudo-label to all unlabeled data. This is error-prone since cluster shapes for the entire dataset are unknown and hardly predictable in a few shot setting. Consequently, it is much harder to learn a discriminate hyperplane when training the downstream intent detection models since some data are incorrectly labeled. Worse still, this grows exponentially as the number of intents (classes) increases in practice since it is harder to predict correct pseudo-labels, and the hyperplane is more difficult to learn since it requires more discriminative power. A corollary is also …

Content truncated to 500 words