Phrase-Level Action Reinforcement Learning for Neural Dialog Response Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5028–5038 August 1–6, 2021. ©2021 Association for Computational Linguistics 5028 Phrase-Level Action Reinforcement Learning for Neural Dialog Response Generation Takato Yamazaki The University of Tokyo, Japan takato@yamazaki.dev Akiko Aizawa The University of Tokyo, Japan National Institute of Informatics, Japan aizawa@nii.ac.jp Abstract Defining a sophisticated action space for a dia- log agent is essential for efficient training with reinforcement learning (RL). Recent work in- troduces discrete latent variables to use as an action space; however, a limitation is that a global vector can contain entangled informa- tion such as dialog act, sentence structure, and content. This sacrifices the flexibility of the response generation. In this paper, we pro- pose phrase-level action reinforcement learn- ing (PHRASERL), which allows the model to flexibly alter the sentence structure and con- tent with the sequential action selection. Our model first learns to generate useful phrases during the supervised pre-training, and then further trained to form a response by rearrang- ing the phrases with reinforcement learning. Experiments on the MultiWOZ dataset show that our model achieves competitive results with state-of-the-art models on automatic eval- uation metrics, indicating that our phrase-level action space has improved flexibility and is ef- fective for solving task-oriented dialogs. 1 Introduction Dialog policy optimization is key research to ef- ficiently solving real-world tasks (Rastogi et al., 2020; Budzianowski et al., 2018; Lewis et al., 2017). In neural response generation, which has made remarkable progress in recent years (Vinyals and Le, 2015; Li et al., 2016a; Serban et al., 2017; Bao et al., 2019), many methods that apply rein- forcement learning (RL) have been proposed (Li et al., 2016b; Peng et al., 2018; Saleh et al., 2019; Zhao et al., 2019). In those studies, one major issue was how to define an action space. Early research proposed a method in which each word of the re- sponse is an action (Li et al., 2016b). However, this has a shortcoming that the generated responses deviate from natural human language (Zhao et al., 2019). A possible reason is that the action space is I recommend area sure , do you prefer I have type of food several options [restaurant_name] is a preference for what would you like to eat … ? hi , a table for you Phrase-Level Action Space sure , what do you prefer type of food ? Hi, can you recommend me an expensive restaurant in the town? User Utterance System Response Figure 1: Demonstration of phrase-level action. The model generates useful phrases and rearrange them to form a response. huge, making it difficult to optimize with RL. More- over, rewarding only the task accomplishment can cause biased improvement, which leads the model to ignore the comprehensibility of the generated response (Wang et al., 2020a). To overcome such issues, LaRL (Zhao et al., 2019) was proposed, which used a discrete global vector to represent dialog acts. In this method, rein- forcement learning is performed only on those dis- crete latent variables, thus …

Content truncated to 500 words