Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection

Proceedings of Recent Advances in Natural Language Processing, pages 480–493 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_054 480 Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection Momchil Hardalov1 Ivan Koychev1 Preslav Nakov2 1Sofia University “St. Kliment Ohridski”, Bulgaria, 2Mohamed bin Zayed University of Artificial Intelligence, UAE {hardalov, koychev}@fmi.uni-sofia.bg preslav.nakov@mbzuai.ac.ae Abstract Detecting the user’s intent and finding the corre- sponding slots among the utterance’s words are important tasks in natural language understand- ing. Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and special- ized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such models, and we design a novel architec- ture on top of them. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token repre- sentations. The experimental results on stan- dard datasets show that our model outperforms both the current non-BERT state of the art as well as stronger BERT-based baselines. 1 Introduction With the proliferation of portable devices, smart speakers, and the evolution of personal assistants, such as Amazon Alexa, Apple Siri, Google Assis- tant, a need for better natural language understand- ing (NLU) has emerged. Moreover, many Web platforms and applications that interact with the users depend on the abilities of an internal NLU component, e.g., customer service with social me- dia (Huang et al., 2021), in dialogue systems in general (Zeng et al., 2021), for web queries under- standing (Tsur et al., 2016; Ye et al., 2016), and general understanding of natural language interac- tion (Vedula et al., 2020). The major challenges such systems face are (i) finding the intention be- hind the user’s request, and (ii) gathering the nec- essary information to complete it via slot filling, while (iii) engaging in a dialogue with the user. Intent PlayMusic Words play music from 2005 by justin broadrick ↓ ↓ ↓ ↓ ↓ ↓ ↓ Slots O O O B-year O B-artist I-artist Table 1: Example from the SNIPS dataset with slots encoded in the BIO format. The utterance’s intent is PlayMusic, and the given slots are year and artist. Table 1 shows a user request collected from a per- sonal voice assistant. Here, the intent is to play music by the artist Justin Broadrick from year 2005. The slot filling task naturally arises as a sequence tagging task. Conventional neural network archi- tectures, such as RNNs or CNNs are appealing approaches to tackle this problem. Various exten- sions thereof can be found in previous work (Xu and Sarikaya, 2013a; Goo et al., 2018; Hakkani- T¨ur et al., 2016; Liu and Lane, 2016; E et al., 2019; Gangadharaiah and Narayanaswamy, 2019). More- over, sequence tagging approaches such as Maxi- mum Entropy Markov model (MEMM) (Toutanvoa and Manning, …

Content truncated to 500 words