Conversational Graph Grounded Policy Learning for Open-Domain Conversation Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1835–1845 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1835 Conversational Graph Grounded Policy Learning for Open-Domain Conversation Generation Jun Xu1∗, Haifeng Wang2, Zheng-Yu Niu2, Hua Wu2, Wanxiang Che1†, Ting Liu1 1Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China 2Baidu Inc., Beijing, China {jxu, car, tliu}@ir.hit.edu.cn, {wanghaifeng, niuzhengyu, wu hua}@baidu.com Abstract To address the challenge of policy learning in open-domain multi-turn conversation, we pro- pose to represent prior information about di- alog transitions as a graph and learn a graph grounded dialog policy, aimed at fostering a more coherent and controllable dialog. To this end, we first construct a conversational graph (CG) from dialog corpora, in which there are vertices to represent “what to say” and “how to say”, and edges to represent natural transi- tion between a message (the last utterance in a dialog context) and its response. We then present a novel CG grounded policy learning framework that conducts dialog flow planning by graph traversal, which learns to identify a what-vertex and a how-vertex from the CG at each turn to guide response generation. In this way, we effectively leverage the CG to facili- tate policy learning as follows: (1) it enables more effective long-term reward design, (2) it provides high-quality candidate actions, and (3) it gives us more control over the policy. Re- sults on two benchmark corpora demonstrate the effectiveness of this framework. 1 Introduction How to effectively learn dialog strategies is an en- during challenge for open-domain multi-turn con- versation generation. To address this challenge, previous works investigate word-level policy mod- els that simultaneously learn dialog policy and lan- guage generation from dialog corpora (Li et al., 2016b; Zhang et al., 2018b). But these word-level policy models often lead to a degeneration issue where the utterances become ungrammatical or repetitive (Lewis et al., 2017). To alleviate this issue, utterance-level policy models have been pro- posed to decouple policy learning from response generation, and they focus on how to incorporate ∗This work was done at Baidu. †Corresponding author: Wanxiang Che. 今天晚上要 通宵 加班 I have to work overnight tonight. 辛苦了，好辛苦，注意身体 Take care of yourself when doing a very hard work. 还不能 打盹，领导也在 I can’t take a nap yet, as the leaders are also here. 这么晚了，不 犯困 啊 ？ It's so late. Don’t you get sleepy? 哈哈，那也会 犯困 吧 Ha-ha, that will make you sleepy. 我以为你会 犯困 的， 这么晚了 I thought you’d be sleepy, as it's late. Context Mechanisms Responses 犯困/sleepy + Message Response Figure 1: Our system (1) understands the user message by linking it to CG. We call the linked vertices as hit what-vertices (green color) ; (2) selects a what-vertex (“sleepy”) and a how-vertex (responding mechanism M3, a MLP network) from one-hop neighbors of hit vertices; (3) generates a coherent response with two sub-steps: firstly, obtains a response representation ¯r using both M3 and a message representation (from a message-encoder); Next, produces a …

Content truncated to 500 words