Unknown Intent Detection using Multi-Objective Optimization on Deep Learning Classifiers

Unknown Intent Detection using Multi-Objective Optimization on Deep Learning Classifiers Prerna Prem1, Zishan Ahmad1, Asif Ekbal1 Shubhashis Sengupta2, Sakshi Jain2, Roshini Rammani2 1Indian Institute of Technology Patna 2Accenture prernaprem21@gmail.com zeeman.zishan@gmail.com asif@iitp.ac.in shubhashis.sengupta@accenture.com sakshi.c.jain@accenture.com roshni.r.ramnani@accenture.com Abstract Modelling and understanding dialogues in a conversation depends on identifying the user intent from the given text. Unknown or new intent detection has became an important task, as in a realistic scenario a user intent may fre- quently change over time and divert even to an intent previously not encountered. This task of separating the unknown intent samples from known intents, is challenging as the un- known user intent can range from intents sim- ilar to the known ones or to something com- pletely different. Prior research for intent dis- covery often considers this problem as a clas- sification task where an unknown intent can belong to a finite or predefined set of known intent classes, thus limiting the scope of such research. In this paper, we tackle the prob- lem of detecting a completely unknown in- tent without any prior hints about the kind of classes belonging to unknown intents. We propose an effective post-processing method using multi-objective optimization to tune an existing neural network based intent classifier making it capable of detecting unknown in- tents. Thus our method can be plugged into existing deep-learning based classifiers and fine-tuned for the purpose of unknown intent detection. We perform experiments using ex- isting state-of-the-art intent classifiers and use our method on top of them for unknown intent detection. Our experiments across different domains and real-world datasets show that our method yields significant improvement com- pared with the state-of-the-art methods for un- known intent detection. 1 Introduction Detecting whether an intent is unknown or new in a dialogue system has become an important task for improving customer satisfaction. Since user intent may frequently change over time in many realistic scenarios, unknown (new) intent detection has be- come an essential problem so, solving which can enhance system interaction with the customer. This task is challenging since there is no prior knowledge of the type or the exact numbers of unknown intents that would be encountered in the future. We model unknown intent detection as an (m+1)- class classification task as suggested by (Shu et al., 2017; Lin and Xu, 2019; Zhang et al., 2020) and group unknown classes into the (m+1)th class. We aim to identify the known intent samples accurately, while at the same time finding the unknown intent samples without having prior knowledge about the unknown intents. Recently, to solve this problem, researchers have used deep neural networks for open classification. OpenMax (Bendale and Boult, 2016) fits Weibull distribution to the outputs of the penulti- mate layer, but requires negative samples for select- ing the best hyperparameters. The MSP (Hendrycks and Gimpel, 2016) calculates the softmax probabil- ity of known samples and discards the low confi- dence unknown samples with the threshold. In our approach, we attempt to solve the problem of un- known intent …

Content truncated to 500 words