Assessing Dialog System User Simulation Evaluation Measures Using Human Judges

Proceedings of ACL-08: HLT, pages 622–629, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Assessing Dialog System User Simulation Evaluation Measures Using Human Judges Hua Ai University of Pittsburgh Pittsburgh PA, 15260, USA hua@cs.pitt.edu Diane J. Litman University of Pittsburgh Pittsburgh, PA 15260, USA litman@cs.pitt.edu Abstract Previous studies evaluate simulated dialog corpora using evaluation measures which can be automatically extracted from the dialog systems’ logs. However, the validity of these automatic measures has not been fully proven. In this study, we first recruit human judges to assess the quality of three simulated dia- log corpora and then use human judgments as the gold standard to validate the conclu- sions drawn from the automatic measures. We observe that it is hard for the human judges to reach good agreement when asked to rate the quality of the dialogs from given perspec- tives. However, the human ratings give con- sistent ranking of the quality of simulated cor- pora generated by different simulation mod- els. When building prediction models of hu- man judgments using previously proposed au- tomatic measures, we find that we cannot reli- ably predict human ratings using a regression model, but we can predict human rankings by a ranking model. 1 Introduction User simulation has been widely used in different phases in spoken dialog system development. In the system development phase, user simulation is used in training different system components. For example, (Levin et al., 2000) and (Scheffler, 2002) exploit user simulations to generate large corpora for using Reinforcement Learning to develop dia- log strategies, while (Chung, 2004) implement user simulation to train the speech recognizer and under- standing components. While user simulation is considered to be more low-cost and time-efficient than experiments with human subjects, one major concern is how well the state-of-the-art user simulations can mimic human user behaviors and how well they can substitute for human users in a variety of tasks. (Schatzmann et al., 2005) propose a set of evaluation measures to assess the quality of simulated corpora. They find that these evaluation measures are sufficient to discern simulated from real dialogs. Since this multiple-measure approach does not offer a easily reportable statistic indicating the quality of a user simulation, (Williams, 2007) proposes a single mea- sure for evaluating and rank-ordering user simula- tions based on the divergence between the simulated and real users’ performance. This new approach also offers a lookup table that helps to judge whether an observed ordering of two user simulations is statisti- cally significant. In this study, we also strive to develop a prediction model of the rankings of the simulated users’ per- formance. However, our approach use human judg- ments as the gold standard. Although to date there are few studies that use human judges to directly as- sess the quality of user simulation, we believe that this is a reliable approach to assess the simulated corpora as well as an important step towards devel- oping a comprehensive set of user simulation evalu- ation measures. First, we can …

Content truncated to 500 words