Exploiting Discourse Structure for Spoken Dialogue Performance Analysis

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 85–93, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Exploiting Discourse Structure for Spoken Dialogue Performance Analysis Mihai Rotaru University of Pittsburgh Pittsburgh, USA mrotaru@cs.pitt.edu Diane J. Litman University of Pittsburgh Pittsburgh, USA litman@cs.pitt.edu Abstract In this paper we study the utility of dis- course structure for spoken dialogue per- formance modeling. We experiment with various ways of exploiting the discourse structure: in isolation, as context infor- mation for other factors (correctness and certainty) and through trajectories in the discourse structure hierarchy. Our corre- lation and PARADISE results show that, while the discourse structure is not useful in isolation, using the discourse structure as context information for other factors or via trajectories produces highly predic- tive parameters for performance analysis. 1 Introduction Predictive models of spoken dialogue system (SDS) performance are an important tool for re- searchers and practitioners in the SDS domain. These models offer insights on what factors are important for the success of a SDS and allow researchers to assess the performance of future system improvements without running additional costly user experiments. One of the most popular models of perform- ance is the PARADISE framework proposed by (Walker et al., 2000). In PARADISE, a set of interaction parameters are measured in a SDS corpus, and then used in a multivariate linear regression to predict the target performance met- ric. A critical ingredient in this approach is the relevance of the interaction parameters for the SDS success. A number of parameters that measure the dialogue efficiency (e.g. number of system/user turns, task duration) and the dia- logue quality (e.g. recognition accuracy, rejec- tions, helps) have been shown to be successful in (Walker et al., 2000). An extensive set of pa- rameters can be found in (Möller, 2005a). In this paper we study the utility of discourse structure as an information source for SDS per- formance analysis. The discourse structure hier- archy has been shown to be useful for other tasks: understanding specific lexical and pro- sodic phenomena (Hirschberg and Nakatani, 1996; Levow, 2004), natural language generation (Hovy, 1993), predictive/generative models of postural shifts (Cassell et al., 2001), and essay scoring (Higgins et al., 2004). We perform our analysis on a corpus of speech-based tutoring dialogues. A tutoring SDS (Litman and Silliman, 2004; Pon-Barry et al., 2004) has to discuss concepts, laws and relation- ships and to engage in complex subdialogues to correct student misconceptions. As a result, dia- logues with such systems have a rich discourse structure. We perform three experiments to measure three ways of exploiting the discourse structure. In our first experiment, we test the predictive utility of the discourse structure in itself. For ex- ample, we look at whether the number of pop-up transitions in the discourse structure hierarchy predicts performance in our system. The second experiment measures the utility of the discourse structure as contextual information for two types of student states: correctness and certainty. The intuition behind this …

Content truncated to 500 words