Structured Dialogue Policy with Graph Neural Networks

Proceedings of the 27th International Conference on Computational Linguistics, pages 1257–1268 Santa Fe, New Mexico, USA, August 20-26, 2018. 1257 Structured Dialogue Policy with Graph Neural Networks Lu Chen∗, Bowen Tan∗, Sishan Long and Kai Yu Key Lab. of Shanghai Education Commission for Intelligent Interaction and Cognitive Eng. SpeechLab, Department of Computer Science and Engineering Brain Science and Technology Research Center Shanghai Jiao Tong University, Shanghai, China {chenlusz, tanbowen, longsishan, kai.yu}@sjtu.edu.cn Abstract Recently, deep reinforcement learning (DRL) has been used for dialogue policy optimization. However, many DRL-based policies are not sample-efficient. Most recent advances focus on improving DRL optimization algorithms to address this issue. Here, we take an alternative route of designing neural network structure that is better suited for DRL-based dialogue management. The proposed structured deep reinforcement learning is based on graph neural networks (GNN), which consists of some sub-networks, each one for a node on a directed graph. The graph is defined according to the domain ontology and each node can be considered as a sub-agent. Dur- ing decision making, these sub-agents have internal message exchange between neighbors on the graph. We also propose an approach to jointly optimize the graph structure as well as the parameters of GNN. Experiments show that structured DRL significantly outperforms previous state-of-the-art approaches in almost all of the 18 tasks of the PyDial benchmark. 1 Introduction A task-oriented spoken dialogue system (SDS) is a system that can continuously interact with a human to accomplish a predefined task, e.g. finding a restaurant or booking a flight. Dialogue management (DM) is the core of an SDS. It has two missions: one is to maintain the dialogue state, and another is to decide how to respond according to a dialogue policy. In this paper, we focus on the dialogue policy. A dialogue policy can be viewed simply as a set of hand-crafted mapping rules from dialogue states to dialogue actions. This is referred to as a rule-based policy. However, in real-world scenarios, un- predictable user behavior, inevitable automatic speech recognition, and spoken language understanding errors make it difficult to maintain the true dialogue state and make the decision. Hence, in recent years, there is a research trend towards statistical dialogue management. A well-founded theory for this is the partially observable Markov decision process (POMDP) (Kaelbling et al., 1998). Under the POMDP-based framework, a distribution of possible states – belief state b is maintained in every dialogue turn. Then reinforcement learning (RL) methods automatically optimize the policy π, i.e. a mapping function from belief state b to dialogue action a = π(b) (Young et al., 2013). Initially, linear RL-based models are adopted, e.g. least-squares policy iteration (LSPI) and natural actor-critic (NAC). However, these linear models have a poor ability of expression and suffer from slow training. Recently, nonparametric algorithms, e.g. Gaussian process reinforcement learning (GPRL), which can optimize policies from a minimal number of dialogues have been proposed. However, the computation cost of GPRL increases with the increase of the number of data. It is therefore …

Content truncated to 500 words