Self-Attention Networks for Intent Detection

Proceedings of Recent Advances in Natural Language Processing, pages 1373–1379, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_157 1373 Self-Attention Networks for Intent Detection Sevinj Yolchuyeva, Géza Németh, Bálint Gyires-Tóth Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics, Budapest, Hungary {syolchuyeva, nemeth, toth.b}@tmit.bme.hu Abstract Self-attention networks (SAN) have shown promising performance in various Natural Lan- guage Processing (NLP) scenarios, especially in machine translation. One of the main points of SANs is the strength of capturing long-range and multi-scale dependencies from the data. In this paper, we present a novel intent detection system which is based on a self-attention net- work and a Bi-LSTM. Our approach shows im- provement by using a transformer model and deep averaging network-based universal sen- tence encoder compared to previous solutions. We evaluate the system on Snips, Smart Speaker, Smart Lights, and ATIS datasets by different evaluation metrics. The performance of the proposed model is compared with LSTM with the same datasets. 1 Introduction and Related Work Spoken dialogue systems are agents that are in- tended to help users to access information effi- ciently by speech interactions (Liu, et al., 2006). In doing so, spoken dialogue systems categorize most of the major fields of spoken language technology, including speech recognition and speech synthesis, language processing, and dialogue system (McTear, 2002). There are different areas of re- search in the field of spoken dialogue systems. Spoken language understanding (SLU) is one of the essential components of spoken dialogue sys- tems, and it aims to form a semantic frame that captures the semantics of user utterances or que- ries. Intent detection is one of the main tasks of SLU system. It can be treated as a semantic utter- ance classification task; since the dialogue system is created to answer a limited range of questions, there is a predefined finite set of intents (Balodis and Deksne, 2019). This task focuses on classify- ing the user’s intent and extracting semantic con- cepts as constraints for natural language. For ex- ample, the utterance “Switch off the garage lights” is related to switching the light off, as shown in Ta- ble 1. Table 1. Example of utterance and a correspond- ing intent label. Utterance Intent Switch off the garage lights. SwitchLightOff Get the room brighter, please. IncreaseBrightness Skip this song and go on to the next one. NextSong Intent detection has been an ongoing field of re- search in SLU, and similar to most NLP tasks, there are two main approaches to identify the intent of an utterance: rule-based and statistical methods (Hashemi, et al., 2016). The rule-based systems use predefined rules to match new utterances to their intents, and these rules need to be carefully engineered by human experts. Thus, the advance- ment of these systems requires a huge amount of human effort. Statistical models, like conditional random field (CRF) and Support Vector Machines, were inves- tigated for this task (Mendoza and Zamora, 2009; Chen et al., 2018). Another important task of SLU is slot filling, which can be …

Content truncated to 500 words