Decay-Function-Free Time-Aware Attention to Context and Speaker Indicator for Spoken Language Understanding

Proceedings of NAACL-HLT 2019, pages 3718–3726 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3718 Decay-Function-Free Time-Aware Attention to Context and Speaker Indicator for Spoken Language Understanding Jonggu Kim Computer Science and Engineering, Pohang University of Science and Technology (POSTECH) Pohang, Republic of Korea jgkimi@postech.ac.kr Jong-Hyeok Lee Computer Science and Engineering, Pohang University of Science and Technology (POSTECH) Pohang, Republic of Korea jhlee@postech.ac.kr Abstract To capture salient contextual information for spoken language understanding (SLU) of a di- alogue, we propose time-aware models that automatically learn the latent time-decay func- tion of the history without a manual time- decay function. We also propose a method to identify and label the current speaker to im- prove the SLU accuracy. In experiments on the benchmark dataset used in Dialog State Tracking Challenge 4, the proposed models achieved significantly higher F1 scores than the state-of-the-art contextual models. Finally, we analyze the effectiveness of the introduced models in detail. The analysis demonstrates that the proposed methods were effective to improve SLU accuracy individually. 1 Introduction Spoken language understanding (SLU) is a com- ponent that understands the user’s utterance of a dialogue system. Given an utterance, SLU gen- erates a structured meaning representation of the utterance; i.e., a semantic frame. SLU can be de- composed into several subtasks such as domain identification, intent prediction and slot filling; these subtasks can be jointly assigned using a single model (Hakkani-T¨ur et al., 2016; Liu and Lane, 2016; Chen et al., 2016b). The accuracy of SLU is important for the dialogue system to gen- erate an appropriate response to a user. To improve the accuracy of SLU, much work has used contextual information of dialogues to al- leviate the ambiguity of recognition of the given utterance. In SLU, selecting important history in- formation is crucial, and it directly influences the improvement of SLU accuracy. To summarize this history, content-aware models (Chen et al., 2016a; Kim et al., 2017) similar to attention models in machine translation (Bahdanau et al., 2014) have been proposed. However, content-aware models are likely to select the wrong history when the histories are similar in content. To alleviate this problem, time-aware models (Chen et al., 2017; Su et al., 2018a,b) which pay attention to recent previous utterances by using the temporal distance between a previous utterance and a current utter- ance are being considered; the models are based on mathematical formulas, time-decay functions, which are formulated by human, and decomposed into trainable parameters. However, the previous time-aware models may not be sufficiently accurate. In the models, ei- ther a single time-decay function is used or a lim- ited number of time-decay functions are linearly combined; these manual functions may not be suf- ficiently flexible to learn an optimal time-decay function. In this paper, we propose flexible and effective time-aware attention models to improve SLU ac- curacy. The proposed models do not need any manual time-decay function, but learn a time- decay tendency directly by introducing a trainable distance vector, and therefore …

Content truncated to 500 words