Setting Up User Action Probabilities in User Simulations for Dialog System Development

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 888–896, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Setting Up User Action Probabilities in User Simulations for Dialog System Development Hua Ai University of Pittsburgh Pittsburgh PA, 15260, USA hua@cs.pitt.edu Diane Litman University of Pittsburgh Pittsburgh PA, 15260, USA litman@cs.pitt.edu Abstract User simulations are shown to be useful in spoken dialog system development. Since most current user simulations deploy prob- ability models to mimic human user be- haviors, how to set up user action proba- bilities in these models is a key problem to solve. One generally used approach is to estimate these probabilities from human user data. However, when building a new dialog system, usually no data or only a small amount of data is available. In this study, we compare estimating user proba- bilities from a small user data set versus handcrafting the probabilities. We discuss the pros and cons of both solutions for dif- ferent dialog system development tasks. 1 Introduction User simulations are widely used in spoken di- alog system development. Recent studies use user simulations to generate training corpora to learn dialog strategies automatically ((Williams and Young, 2007), (Lemon and Liu, 2007)), or to evaluate dialog system performance (L´opez-C´ozar et al., 2003). Most studies show that using user simulations significantly improves dialog system performance as well as speeds up system devel- opment. Since user simulation is such a useful tool, dialog system researchers have studied how to build user simulations from a variety of perspec- tives. Some studies look into the impact of training data on user simulations. For example, (Georgila et al., 2008) observe differences between simu- lated users trained from human users of different age groups. Other studies explore different simu- lation models, i.e. the mechanism of deciding the next user actions given the current dialog context. (Schatzmann et al., 2006) give a thorough review of different types of simulation models. Since most of these current user simulation techniques use probabilistic models to generate user actions, how to set up the probabilities in the simulations is another important problem to solve. One general approach to set up user action prob- abilities is to learn the probabilities from a col- lected human user dialog corpus ((Schatzmann et al., 2007b), (Georgila et al., 2008)). While this approach takes advantage of observed user behav- iors in predicting future user behaviors, it suffers from the problem of learning probabilities from one group of users while potentially using them with another group of users. The accuracy of the learned probabilities becomes more questionable when the collected human corpus is small. How- ever, this is a common problem in building new dialog systems, when often no data1 or only a small amount of data is available. An alterna- tive approach is to handcraft user action proba- bilities ((Schatzmann et al., 2007a), (Janarthanam and Lemon, 2008)). This approach is less data- intensive, but requires nontrivial work by domain experts. What is …

Content truncated to 500 words