Modeling with Recurrent Neural Networks for Open Vocabulary Slots

Proceedings of the 27th International Conference on Computational Linguistics, pages 2778–2790 Santa Fe, New Mexico, USA, August 20-26, 2018. 2778 Modeling with Recurrent Neural Networks for Open Vocabulary Slots Jun-Seong Kim Samsung Electronics Co., Ltd / Suwon, South Korea Js087.kim@samsung.com Junghoe Kim, SeongUn Park, Kwangyong Lee, Yoonju Lee Samsung Electronics Co., Ltd / Suwon, South Korea {kjh94, seongun.park, ky4you.lee, yo0nju.lee}@samsung.com Abstract Dealing with ‘open-vocabulary’ slots has been among the challenges in the natural language area. While recent studies on attention-based recurrent neural network (RNN) models have performed well in completing several language related tasks such as spoken language under- standing and dialogue systems, there has been a lack of attempts to address filling slots that take on values from a virtually unlimited set. In this paper, we propose a new RNN model that can capture the vital concept: Understanding the role of a word may vary according to how long a reader focuses on a particular part of a sentence. The proposed model utilizes a long- term aware attention structure, positional encoding primarily considering the relative distance between words, and multi-task learning of a character-based language model and an intent de- tection model. We show that the model outperforms the existing RNN models with respect to discovering ‘open-vocabulary’ slots without any external information, such as a named entity database or knowledge base. In particular, we confirm that it performs better with a greater number of slots in a dataset, including unknown words, by evaluating the models on a dataset of several domains. In addition, the proposed model also demonstrates superior performance with regard to intent detection. 1 Introduction In spoken dialogue systems and goal-oriented dialogue systems, slot filling and intent detection are primarily involved in a process of understanding user utterances based on pre-designed semantic frames. Slot filling is to identify a sequence of tokens and extract semantic constituents from the ut- terances, and intent detection is to classify the intent of the utterances. Two examples relevant to these problems are shown in Figure 1 for the domains of flight and message. In recent years, a significant amount of interest has built up around slot filling and intent detection due in part to the competition among commercial artificial intelligence assistants, such as Samsung Bixby, Apple Siri, Google Assistant, Microsoft Cortana, and Amazon Alexa. As a result, a multitude of studies regarding recurrent neural network (RNN) models for slot filling and intent detection has been completed and has resulted in outcomes that generally outperform most other past approaches. In par- ticular, there have been various attempts to improve the performance of RNNs for spoken dialogue systems such as label sampling (Liu and Lane, 2015), the hybrid type of Elman and Jordan (Mesnil et al,. 2015), Deep LSTM (Yao et al., 2014), external memory (Peng et al., 2015), bidirectional LSTM (Hakkani-Tur et al., 2016), encoder-labeler and Deep LSTM (Kurata et al., 2016), attention (Liu and Lane, 2016), bidirectional LSTM and CRF (Huang et al., 2015), word hashing (Ravuri and Stolcke, 2015a; Ravuri …

Content truncated to 500 words