High-Quality Dialogue Diversification by Intermittent Short Extension Ensembles

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1861–1872 August 1–6, 2021. ©2021 Association for Computational Linguistics 1861 High-Quality Diversification for Task-Oriented Dialogue Systems Zhiwen Tang Hrishikesh Kulkarni Grace Hui Yang InfoSense, Department of Computer Science Georgetown University {zt79,hpk8,grace.yang}@georgetown.edu Abstract Many task-oriented dialogue systems use deep reinforcement learning (DRL) to learn policies that respond to the user appropriately and com- plete the tasks successfully. Training DRL agents with diverse dialogue trajectories pre- pare them well for rare user requests and un- seen situations. One effective diversification method is to let the agent interact with a di- verse set of learned user models. However, tra- jectories created by these artificial user mod- els may contain generation errors, which can quickly propagate into the agent’s policy. It is thus important to control the quality of the diversification and resist the noise. In this paper, we propose a novel dialogue diversifi- cation method for task-oriented dialogue sys- tems trained in simulators. Our method, In- termittent Short Extension Ensemble (I-SEE),1 constrains the intensity to interact with an en- semble of diverse user models and effectively controls the quality of the diversification. Eval- uations on the Multiwoz dataset show that I- SEE successfully boosts the performance of several state-of-the-art DRL dialogue agents. 1 Introduction Task-oriented dialogue agents assist human users to complete their tasks in multi-round human-agent in- teractions. Example tasks include booking a movie ticket or reserving a lunch table. Many agents use deep reinforcement learning (DRL) to learn good policies that respond appropriately in the dialogue and succeed in completing the task (Schulman et al., 2017; Takanobu et al., 2020; Peng et al., 2018). Due to the high cost of interactive train- ing using real human users, simulators have been used to replace the humans (Shi et al., 2019). Early user simulators rely on rules hand-crafted by do- main experts, who review and summarize dialogue 1I-SEE codes can be found at https://github.com/ smt-HS/I-SEE. Figure 1: Agent Learns from Diversified Simulators. templates from everyday conversations (Li et al., 2016b). Rule-based simulators manage routine task scenarios efficiently; but they cannot easily gen- erate unconventional and more diverse dialogue acts marked by human spontaneity. Agents learn- ing from these simulators may fail to handle rare queries and unseen situations. For example, in Table 1’s left column, the system learns from a rule-based simulator and fails to answer an unex- pected question “In which area of the city is this hotel located?” The user immediately ends the con- versation and does not book the hotel. Increasing dialogue diversity is a long-lasting re- search interest motivated by different needs. In dia- logue generation, diversity is desired to avoid dull responses. In simulator construction and reinforce- ment learning (RL), diversity is desired for training robust agents. Although the uses are slightly dif- ferent, ideas to improve diversification can be uni- versal. Dialogue diversity can be improved via i) enforcing diversity in objective functions (such as maximize mutual information) of neural models (Li et al., 2016a; Baheti et …

Content truncated to 500 words