Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3952–3961, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3952 Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes Paulo Cavalin, Victor Henrique Alves Ribeiro, Ana Paula Appel, Claudio Pinhanez IBM Research S˜ao Paulo, SP, Brazil pcavalin@br.ibm.com Abstract This paper explores how intent classification can be improved by representing the class la- bels not as a discrete set of symbols but as a space where the word graphs associated to each class are mapped using typical graph em- bedding techniques. The approach, inspired by a previous algorithm used for an inverse dictionary task, allows the classification algo- rithm to take in account inter-class similarities provided by the repeated occurrence of some words in the training examples of the differ- ent classes. The classification is carried out by mapping text embeddings to the word graph embeddings of the classes. Focusing solely on improving the representation of the class label set, we show in experiments conducted in both private and public intent classification datasets, that better detection of out-of-scope examples (OOS) is achieved and, as a conse- quence, that the overall accuracy of intent clas- sification is also improved. In particular, using the recently-released Larson dataset, an error of about 9.9% has been achieved for OOS de- tection, beating the previous state-of-the-art re- sult by more than 31 percentage points. 1 Introduction Intent classification is usually applied for response selection in conversational systems, such as text- based chatbots. For the end-user to have the best possible experience with those systems, it is ex- pected that an intent classifier is able not only to map an input utterance to the correct intent but also to detect when the utterance is not related to any of the intents, to which we refer to as out-of- scope (OOS)1 inputs or samples. In the light of this, this paper describes and evaluates a method which tries to capture the complexity of the set of intents by embedding them into a vector space 1Out-of-domain examples is also a common term in the literature. created using word graphs, as described later. We show that, although the method in some cases is able to improve the accuracy of a text classifier in in-scope examples, it has often a tremendous im- pact on improving the ability of text classifier to reject OOS text, without relying on OOS examples in the training set. Notice that the intent classifier is typically im- plemented using standard text classification algo- rithms (Weiss et al., 2012; Larson et al., 2019; Casanueva et al., 2020). Consequently, to per- form OOS sample detection, methods often rely on one-class classification or threshold rejection- based techniques using the probability outputs for each class (Larson et al., 2019) or reconstruction errors (Ryu et al., 2017, 2018). There also exist approaches based on the assump- tion that OOS data can be collected and included in the training set (Tan et …

Content truncated to 500 words