Learning Mixed Initiative Dialog Strategies By Using Reinforcement Learning On Both Conversants

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 1011–1018, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Learning Mixed Initiative Dialog Strategies By Using Reinforcement Learning On Both Conversants Michael S. English and Peter A. Heeman Center for Spoken Language Understanding OGI School of Science & Engineering Oregon Health & Science University Beaverton OR, 97006, USA menglish6@gmail.com and heeman@cslu.ogi.edu Abstract This paper describes an application of re- inforcement learning to determine a dia- log policy for a complex collaborative task where policies for both the system and a proxy for a user of the system are learned simultaneously. With this approach a use- ful dialog policy is learned without the drawbacks of other approaches that re- quire significant human interaction. The specific task that the agents were trained on was chosen for its complexity and re- quirement that both conversants bring task knowledge to the interaction, thus ensur- ing its collaborative nature. The results of our experiment show that you can use re- inforcement learning to create an effective dialog policy, which employs a mixed ini- tiative strategy, without the drawbacks of large amounts of data or significant human input. 1 Introduction The problem of developing a dialog manager can be expressed as the task of building a specific dialog policy for the dialog system to follow as it interacts with the user. A dialog policy can be thought of as an enumeration of all of the states a dialog system can be in, and the corresponding action to take from each of those states. Thus a policy completely specifies the behavior of a dialog manager. Most conventional approaches to accomplishing this task seek to directly model human interactions in some manner. These techniques include hand- crafting a policy, using a Wizard-of-Oz approach in an iterative manner and inducing a policy from a human-human dialog corpus. All three approaches have shortcomings that make them less than ideal for developing dialog systems. The approach of hand- crafting of a dialog policy is problematic as it is difficult to predict how a user with interact with it, making it difficult to craft an optimal policy. To get around this, an iterative approach can be used, with a Wizard taking the place of the system. However, it is still difficult to train a wizard, and it is difficult to explore many different strategies in order to find the optimal one. Human-human dialog can be used for policy generation, as this should represent optimal behavior to accomplish a task. However, computers are not capable of behaving exactly as a human. In addition, humans might not interact with a computer as they would another person. Recently a number of researchers have proposed using reinforcement learning to alleviating the prob- lems encountered with more conventional methods of developing dialog policies. With the development of a good policy evaluation function, reinforcement learning can effectively and quickly explore a large policy space. There is the additional benefit that it …

Content truncated to 500 words