Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

Proceedings of NAACL-HLT 2019, pages 1208–1218 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1208 Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models Tiancheng Zhao1, Kaige Xie2 and Maxine Eskenazi1 1Language Technologies Institute, Carnegie Mellon University 2Shanghai Jiao Tong University {tianchez, max+}@cs.cmu.edu, lightyear0117@sjtu.edu.cn Abstract Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an en- during challenge. Common practice has been to use handcrafted dialog acts, or the output vocabulary, e.g. in neural encoder decoders, as the action spaces. Both have their own limita- tions. This paper proposes a novel latent action framework that treats the action spaces of an end-to-end dialog agent as latent variables and develops unsupervised methods in order to in- duce its own action space from the data. Com- prehensive experiments are conducted exam- ining both continuous and discrete action types and two different optimization methods based on stochastic variational inference. Results show that the proposed latent actions achieve superior empirical performance improvement over previous word-level policy gradient meth- ods on both DealOrNoDeal and MultiWoz di- alogs. Our detailed analysis also provides in- sights about various latent variable approaches for policy learning and can serve as a founda- tion for developing better latent actions in fu- ture research. 1 1 Introduction Optimizing dialog strategies in multi-turn dialog models is the cornerstone of building dialog sys- tems that more efficiently solve real-world chal- lenges, e.g. providing information (Young, 2006), winning negotiations (Lewis et al., 2017), improv- ing engagement (Li et al., 2016) etc. A clas- sic solution employs reinforcement learning (RL) to learn a dialog policy that models the opti- mal action distribution conditioned on the dialog state (Williams and Young, 2007). However, since there are infinite human language possibilities, an enduring challenge has been to define what the 1Data and code are available at https://github. com/snakeztc/NeuralDialog-LaRL action space is. For traditional modular systems, the action space is defined by hand-crafted seman- tic representations such as dialog acts and slot- values (Raux et al., 2005; Chen et al., 2013) and the goal is to obtain a dialog policy that chooses the best hand-crafted action at each dialog turn. But it is limited because it can only handle simple domains whose entire action space can be captured by hand-crafted representations (Walker, 2000; Su et al., 2017). This cripples a system’s ability to handle conversations in complex domains. Conversely, end-to-end (E2E) dialog systems have removed this limit by directly learning a re- sponse generation model conditioned on the dia- log context using neural networks (Vinyals and Le, 2015; Sordoni et al., 2015). To apply RL to E2E systems, the action space is typically defined as the entire vocabulary; every response output word is considered to be an action selection step (Li et al., 2016), which we denote as the word-level RL. Word-level RL, however, has been shown to have several major limitations in learning dialog strategies. The foremost one is that direct …

Content truncated to 500 words