Single-Agent vs. Multi-Agent Techniques for Concurrent Reinforcement Learning of Negotiation Dialogue Policies

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 500–510, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Single-Agent vs. Multi-Agent Techniques for Concurrent Reinforcement Learning of Negotiation Dialogue Policies Kallirroi Georgila, Claire Nelson, David Traum University of Southern California Institute for Creative Technologies 12015 Waterfront Drive, Playa Vista, CA 90094, USA {kgeorgila,traum}@ict.usc.edu Abstract We use single-agent and multi-agent Rein- forcement Learning (RL) for learning dia- logue policies in a resource allocation ne- gotiation scenario. Two agents learn con- currently by interacting with each other without any need for simulated users (SUs) to train against or corpora to learn from. In particular, we compare the Q- learning, Policy Hill-Climbing (PHC) and Win or Learn Fast Policy Hill-Climbing (PHC-WoLF) algorithms, varying the sce- nario complexity (state space size), the number of training episodes, the learning rate, and the exploration rate. Our re- sults show that generally Q-learning fails to converge whereas PHC and PHC-WoLF always converge and perform similarly. We also show that very high gradually decreasing exploration rates are required for convergence. We conclude that multi- agent RL of dialogue policies is a promis- ing alternative to using single-agent RL and SUs or learning directly from corpora. 1 Introduction The dialogue policy of a dialogue system decides on which actions the system should perform given a particular dialogue state (i.e., dialogue context). Building a dialogue policy can be a challenging task especially for complex applications. For this reason, recently much attention has been drawn to machine learning approaches to dialogue man- agement and in particular Reinforcement Learning (RL) of dialogue policies (Williams and Young, 2007; Rieser et al., 2011; Jurˇc´ıˇcek et al., 2012). Typically there are three main approaches to the problem of learning dialogue policies using RL: (1) learn against a simulated user (SU), i.e., a model that simulates the behavior of a real user (Georgila et al., 2006; Schatzmann et al., 2006); (2) learn directly from a corpus (Henderson et al., 2008; Li et al., 2009); or (3) learn via live interac- tion with human users (Singh et al., 2002; Gaˇsi´c et al., 2011; Gaˇsi´c et al., 2013). We propose a fourth approach: concurrent learning of the system policy and the SU policy using multi-agent RL techniques. Both agents are trained simultaneously and there is no need for building a SU separately or having access to a cor- pus.1 As we discuss below, concurrent learning could potentially be used for learning via live in- teraction with human users. Moreover, for negoti- ation in particular there is one more reason in fa- vor of concurrent learning as opposed to learning against a SU. Unlike slot-filling domains, in nego- tiation the behaviors of the system and the user are symmetric. They are both negotiators, thus build- ing a good SU is as difficult as building a good system policy. So far research on using RL for dialogue pol- icy learning has focused on single-agent RL tech- niques. Single-agent RL methods make the as- …

Content truncated to 500 words