Controlling Listening-oriented Dialogue using Partially Observable Markov Decision Processes

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 761–769, Beijing, August 2010 Controlling Listening-oriented Dialogue using Partially Observable Markov Decision Processes Toyomi Meguro†, Ryuichiro Higashinaka‡, Yasuhiro Minami†, Kohji Dohsaka† †NTT Communication Science Laboratories, NTT Corporation ‡NTT Cyber Space Laboratories, NTT Corporation meguro@cslab.kecl.ntt.co.jp higashinaka.ryuichiro@lab.ntt.co.jp {minami,dohsaka}@cslab.kecl.ntt.co.jp Abstract This paper investigates how to automat- ically create a dialogue control compo- nent of a listening agent to reduce the cur- rent high cost of manually creating such components. We collected a large number of listening-oriented dialogues with their user satisfaction ratings and used them to create a dialogue control component using partially observable Markov decision pro- cesses (POMDPs), which can learn a pol- icy to satisfy users by automatically find- ing a reasonable reward function. A com- parison between our POMDP-based com- ponent and other similarly motivated sys- tems using human subjects revealed that POMDPs can satisfactorily produce a dia- logue control component that can achieve reasonable subjective assessment. 1 Introduction Although task-oriented dialogue systems have been actively researched (Hirshman, 1989; Fer- guson et al., 1996; Nakano et al., 1999; Walker et al., 2002), recently non-task-oriented functions are starting to attract attention, and systems with- out a specific task that deal with more casual di- alogues, such as chats, are being actively investi- gated from their social and entertainment aspects (Bickmore and Cassell, 2001; Higashinaka et al., 2008; Higuchi et al., 2008). In the same vein, we have been working on listening-oriented dialogues in which one conver- sational participant attentively listens to the other (hereafter, listening-oriented dialogue). Our aim is to build listening agents that can implement such a listening process so that users can satisfy their desire to speak and be heard. Figure 1 shows an excerpt from a typical listening-oriented dia- logue. In the literature, dialogue control compo- nents for less (or non-) task-oriented dialogue sys- tems, such as listening agents, have typically used hand-crafted rules for dialogue control, which can be problematic because completely covering all dialogue states by hand-crafted rules is diffi- cult when the dialogue has fewer task restrictions (Wallace, 2004; Isomura et al., 2009). To solve this problem, this paper aims to auto- matically build a dialogue control component of a listening agent using partially observable Markov decision processes (POMDPs). POMDPs, which make it possible to learn a policy that can max- imize the averaged reward in partially observable environments (Pineau et al., 2003), have been suc- cessfully adopted in task-oriented dialogue sys- tems for learning a dialogue control module from data (Williams and Young, 2007). However, no work has attempted to use POMDPs for less (or non-) task-oriented dialogue systems, such as lis- tening agents, because user goals are not as well- defined as task-oriented ones, complicating the finding of a reasonable reward function. We apply POMDPs to listening-oriented dia- logues by having the system learn a policy that si- multaneously maximizes how well users feel that they are being listened to (hereafter, user satis- faction) and how smoothly the system generates dialogues …

Content truncated to 500 words