Counterfactual Off-Policy Training for Neural Dialogue Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3438–3448, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3438 Counterfactual Off-Policy Training for Neural Dialogue Generation Qingfu Zhu♯, Weinan Zhang♯∗, Ting Liu♯, William Yang Wang♭ ♯Harbin Institute of Technology, Harbin, China ♭University of California, Santa Barbara, USA {qfzhu, wnzhang, tliu}@ir.hit.edu.cn william@cs.ucsb.edu Abstract Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this pa- per, we propose to explore potential responses by counterfactual reasoning. Given an ob- served response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthe- sized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses under the ad- versarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model as well as the conventional adversarial learning approaches. 1 Introduction Open-domain dialogue generation (Shang et al., 2015a; Vinyals and Le, 2015; Sordoni et al., 2015a) intends to produce coherent responses given dia- logue history. Nevertheless, it suffers from data insufficiency problem as there may exist many po- tential responses for a given dialogue history (Li et al., 2016). An ideal way of exploring the po- tential responses is to train the model by chatting with real users, which is usually time-consuming and labor-intensive in practice. Although replac- ing a real user with a user simulator could address the issue, the simulator only roughly approximates real user statistics, and its development process is costly (Su et al., 2016). In contrast, humans could independently reason potential responses based on past experiences from the true environment. Having observed a response, one might naturally ask himself or herself: “What ∗Corresponding author. Dialogue History: What are you up to this Friday? Observed Response: Well, I have the day off from work. Counterfactual Response: I am going to gym with a friend. what if I respond differently Figure 1: An example of a counterfactual response, which is a potential response inferred in hindsight from given observed response. would happen if I respond differently, while every- thing else in the environment remains the same.” Answering the question will result in a potential response (as an example in Figure 1), and it is bene- ficial for improving future decision making (Roese, 1997). The potential response inferred in hind- sight is called a counterfactual response, where the concept “counterfactual” describes the posterior process of reasoning the outcome of alternative ac- tions (i.e., a different responding policy) that could have been taken while keeping everything else un- changed (Buesing et al., 2019). Motivated by this, we propose a counterfactual off-policy training (COPT) approach to explore po- tential responses. Building upon the adversarial learning framework, COPT casts a dialogue gen- erator as a structural causal model (SCM), which describes a generation process with two ingredi- ents: scenarios …

Content truncated to 500 words