How to Build User Simulators to Train RL-based Dialog Systems

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1990–2000, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1990 How to Build User Simulators to Train RL-based Dialog Systems Weiyan Shi⇤1, Kun Qian⇤1, Xuewei Wang2 and Zhou Yu1 1 University of California, Davis, 2 Carnegie Mellon University {wyshi, kunqian, joyu}@ucdavis.edu, xueweiwa@andrew.cmu.edu Abstract User simulators are essential for training rein- forcement learning (RL) based dialog models. The performance of the simulator directly im- pacts the RL policy. However, building a good user simulator that models real user behaviors is challenging. We propose a method of stan- dardizing user simulator building that can be used by the community to compare dialog sys- tem quality using the same set of user simu- lators fairly. We present implementations of six user simulators trained with different dia- log planning and generation methods. We then calculate a set of automatic metrics to eval- uate the quality of these simulators both di- rectly and indirectly. We also ask human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems. This paper presents a comprehensive evaluation framework for user simulator study and provides a better under- standing of the pros and cons of different user simulators, as well as their impacts on the trained systems. 1 1 Introduction Reinforcement Learning has gained more and more attention in dialog system training because it treats the dialog planning as a sequential deci- sion problem and focuses on long-term rewards (Su et al., 2017). However, RL requires inter- action with the environment, and obtaining real human users to interact with the system is both time-consuming and labor-intensive. Therefore, building user simulators to interact with the sys- tem before deployment to real users becomes an economical choice (Williams et al., 2017; Li et al., 2016). But the performance of the user simula- tor has a direct impact on the trained RL policy. * Equal contribution. 1The code and data are released at https://github. com/wyshi/user-simulator. Such an intertwined relation between user simula- tor and dialog system makes the whole process a “chicken and egg” problem. This naturally leads to the question of how different user simulators impact the system performance, and how to build appropriate user simulators for different tasks. In previous RL-based dialog system literature, people reported their system’s performance, such as success rate, on their specific user simulators (Liu and Lane, 2017; Shi and Yu, 2018), but the details of the user simulators are not sufficient to reproduce the results. User simulators’ quality can vary in multiple aspects, which could lead to un- fair comparison between different trained systems. For instance, RL systems built with more compli- cated user simulators will have lower scores on the automatic metrics, compared to those built using simpler user simulators. However, the good per- formance may not necessarily transfer when the system is tested by real users. In fact, …

Content truncated to 500 words