Dialog policy optimization for low resource setting using Self-play and Reward based Sampling

Dialogue policy optimization for low resource setting using Self-play and Reward based Sampling Tharindu Madusanka, Durashi Langappuli, Thisara Welmilla, Uthayasanker Thayasivam and Sanath Jayasena Department of Computer Science and Engineering,University of Moratuwa ,Sri Lanka {stharindu.16, durashi.16, welmilla.16, rtuthaya, sanath}@cse.mrt.ac.lk Abstract Reinforcement Learning is considered as the state of the art approach for dialogue policy optimization in task-oriented dialogue systems. However, these models demand a large cor- pus of dialogues to learn effectively. Train- ing Reinforcement Learning agent with low data amount tends to overfit the agent. Al- though synthesizing dialogue agendas with di- alogue Self-play using rule-based agents and crowdsourcing has demonstrated promising re- sults with the low amount of samples, these methods hold limitations. For instance, rule- based agents acquire specific domain and lan- guage while crowdsourcing demands a high price and domain experts, especially in local languages. In this paper, we address these limitations by proposing a novel approach for synthetic agenda generation by acknowledg- ing the underlying probability distribution of the user agendas and a reward-based sampling method that prioritizes failed dialogue acts. Evaluations conducted shows leveraged per- formance without overfitting, compared to the baseline method. Also, the reward-based sam- pling method improves the overall mean task success rate by an average of 11.307%. 1 Introduction A dialogue system, or conversational agent, denotes a system that can conduct a conversation with another agent, usually a human (Perez-Marin and Pascual- Nieto, 2011). One type of conversational agent are Task-oriented conversational agents, that can help users accomplish tasks ranging from meeting schedul- ing to vacation planning. The structure of a task- Figure 1: Components of a task-oriented conversational agent oriented conversational agent is outlined in Figure 1. It consists of (i) a natural language understand- ing(NLU) module for identifying intents of user utter- ances (ii) a dialogue state tracker(DST) for tracking conversation state (iii) a dialogue policy learner(POL) which selects the next action based on the current state (iv) a natural language generator(NLG) for con- verting the agent action to a natural language re- sponse (Gao et al., 2018). Dialogue Policy Network(learner) plays a critical role in task-oriented conversational agents since they require logical reasoning and planning over several dialogue turns. Policy Network has been developed under rule-based methods(i.e. ontology-based, finite state machines) and model-based methods(i.e. Su- pervised Learning(SL) based (Bordes et al., 2016; Dinan et al., 2018), Reinforcement Learning(RL) based (Lu et al., 2019; Liu and Lane, 2018; Su et al., 2016a)). Due to enhanced data availability, dialogue policy optimization has leveraged with RL based methods. However, these state-of-the-art RL meth- ods have barely experimented in the low resource Figure 2: Training and test success rates of user goals with low amount of samples setting because these models tend to overfit when the number of available training data is low(see Figure 2). The accuracy of the dialogue policy network di- rectly depends on the availability of quality dialogue samples to train. As depicts in Figure 2, the agent tends to overfit when the number of training samples is …

Content truncated to 500 words