Unknown Intent Detection Using Multi-Objective Optimization on Deep Learning Classifiers

Proceedings of Recent Advances in Natural Language Processing, pages 1130–1137 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_127 1130 Unknown Intent Detection Using Multi-Objective Optimization on Deep Learning Classifiers Prerna Prem1, Zishan Ahmad1, Asif Ekbal1 Shubhashis Sengupta2, Sakshi C. Jain2, Roshni Ramnani2 1Department of Computer Science and Engineering, Indian Institute of Technology Patna 2Accenture Technology Lab, Bangalore prernaprem21@gmail.com 1821cs18@iitp.ac.in asif.ekbal@gmail.com shubhashis.sengupta@accenture.com sakshi.c.jain@accenture.com roshni.r.ramnani@accenture.com Abstract Modelling and understanding dialogues in a conversation depends on identifying the user intent from the given text. Unknown or new in- tent detection is a critical task, as in a realistic scenario a user intent may frequently change over time and divert even to an intent previ- ously not encountered. This task of separat- ing the unknown intent samples from known intents one is challenging as the unknown user intent can range from intents similar to the pre- defined intents to something completely dif- ferent. Prior research on intent discovery of- ten consider it as a classification task where an unknown intent can belong to a predefined set of known intent classes. In this paper we tackle the problem of detecting a completely unknown intent without any prior hints about the kind of classes belonging to unknown in- tents. We propose an effective post-processing method using multi-objective optimization to tune an existing neural network based intent classifier and make it capable of detecting un- known intents. We perform experiments using existing state-of-the-art intent classifiers and use our method on top of them for unknown in- tent detection. Our experiments across differ- ent domains and real-world datasets show that our method yields significant improvements compared with the state-of-the-art methods for unknown intent detection. 1 Introduction Detecting whether an intent is unknown or new in a dialogue system has become an important task for improving customer satisfaction. Since user intent may frequently change over time in many realistic scenarios, unknown (new) intent detection has become a crucial problem in conversational ar- tificial intelligence (CAI). This can ultimately help enhance system interaction with the customer. This task is challenging since there is no prior knowl- edge of the type or the exact numbers of unknown intents that would be encountered in the future. We model unknown intent detection as an (m+1)- class classification task as suggested by (Shu et al., 2017; Lin and Xu, 2019; Zhang et al., 2020) and consider unknown classes as the (m+1)th class. We aim to identify the known intent samples accurately, while at the same time we focus on determining the unknown intent samples. This has to be done without any prior knowledge about the kind of un- known intents. In order to solve this problem, re- searchers have proposed deep neural networks like OpenMax (Bendale and Boult, 2016), which fits Weibull distribution to the outputs of the penulti- mate layer. Another system MSP (Hendrycks and Gimpel, 2016) calculates the softmax probability of known samples and discards the unknown sam- ples with lower confidence. In our approach, we attempt to solve the problem of unknown intent detection with …

Content truncated to 500 words