Actor-Double-Critic: Incorporating Model-Based Critic for Task-Oriented Dialogue Systems

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 854–863 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 854 Actor-Double-Critic: Incorporating Model-Based Critic for Task-Oriented Dialogue Systems Yen-Chen Wu1, Bo-Hsiang Tseng1, and Milica Gaˇsi´c2 1University of Cambridge, UK {ycw30, bht26}@cam.ac.uk 2Heinrich Heine University D¨usseldorf, Germany gasic@uni-duesseldorf.de Abstract In order to improve the sample-efficiency of deep reinforcement learning (DRL), we im- plemented imagination augmented agent (I2A) in spoken dialogue systems (SDS). Although I2A achieves a higher success rate than base- lines by augmenting predicted future into a policy network, its complicated architecture in- troduces unwanted instability. In this work, we propose actor-double-critic (ADC) to im- prove the stability and overall performance of I2A. ADC simplifies the architecture of I2A to reduce excessive parameters and hyper- parameters. More importantly, a separate model-based critic shares parameters between actions and makes back-propagation explicit. In our experiments on Cambridge Restaurant Booking task, ADC enhances success rates considerably and shows robustness to imper- fect environment models. In addition, ADC exhibits the stability and sample-efficiency as significantly reducing the baseline standard de- viation of success rates and reaching the 80% success rate with half training data. 1 Introduction Spoken Dialogue Systems (SDS) enable human- computer interaction via natural language. The core of SDS, dialogue management, can be formu- lated as an RL problem (Levin et al., 1997; Young et al., 2013; Williams, 2008). Great advancements can be achieved with deep RL algorithms (Dhingra et al., 2016; Chang et al., 2017; Budzianowski et al., 2017; Casanueva et al., 2017; Liu et al., 2018; Gao et al., 2018; Takanobu et al., 2019; Wu et al., 2020). Yet, deep RL methods are notoriously expensive in terms of the number of interactions they require. Even relatively simple tasks can require thousands of labelled dialogues and modelling complex be- haviour such as a multi-domain application might need substantially more (Gaˇsi´c et al., 2011; Li et al., 2016; Su et al., 2016). Model-based reinforcement learning (MBRL) is one way of improving sample-efficiency in RL (Tamar et al., 2016; Silver et al., 2016; Gu et al., 2016; Nagabandi et al., 2018; Oh et al., 2017). By learning the environment model, we can predict the future states after taking a certain action. In a dialogue system, that means the system can predict the user’s behaviour. In contrast, the model-free RL algorithms only learn the mapping of belief states and Q-values and do not make use of the user behaviour patterns in the training data. In other words, model-free RL is wasting actions by going through similar transitions multiple times to get accurate return estimations. Dyna-Q (Sutton, 1990; Sutton et al., 2012) has achieved some success in SDS (Peng et al., 2018; Su et al., 2018; Wu et al., 2019; Zhang et al., 2019) by generating training data for agents and keeping improving its environment model from real inter- actions between agents and users. Nevertheless, the noisy data generated by inaccurate environment models could adversely affect the experience re- play buffer and result …

Content truncated to 500 words