Mining Spoken Dialogue Corpora for System Evaluation and Modelin

Mining Spoken Dialogue Corpora for System Evaluation and Modeling Frederic Bechet LIA-CNRS University of Avignon, France frederic.bechet@ lia.univ-avignon.fr Giuseppe Riccardi AT&T Labs Florham Park, NJ, USA dsp3@ research.att.com Dilek Hakkani-Tur AT&T Labs Florham Park, NJ, USA dtur@ research.att.com Abstract We are interested in the problem of modeling and evaluating spoken language systems in the context of human-machine dialogs. Spoken di- alog corpora allow for a multidimensional anal- ysis of speech recognition and language under- standing models of dialog systems. Therefore language models can be directly trained based either on the dialog history or its equivalence class (or cluster). In this paper we propose an algorithm to mine dialog traces which exhibit similar patterns and are identified by the same class. For this purpose we apply data clustering methods to large human-machine spoken dia- logue corpora. The resulting clusters can be used for system evaluation and language mod- eling. By clustering dialog traces we expect to learn about the behavior of the system with re- gards to not only the automation rate but the nature of the interaction (e.g. easy vs difficult dialogs). The equivalence classes can also be used in order to automatically adapt the lan- guage model, the understanding module and the dialogue strategy to better fit the kind of in- teraction detected. This paper investigates dif- ferent ways for encoding dialogues into multi- dimensional structures and different clustering methods. Preliminary results are given for clus- ter interpretation and dynamic model adapta- tion using the clusters obtained. 1 Introduction The deployment of large scale automatic spoken dialog systems, like How May I Help You?SM (HMIHY) (Gorin et al., 1997), makes avail- able large corpora of real human-machine di- alog interactions. Traditionally, this data is used for supervised system evaluation. For in- stance, in (Kamm et al., 1999) they propose a static analysis aimed at measuring the perfor- mance of a dialog system, especially in an at- tempt to automatically estimate user satisfac- tion. In (Hastie et al., 2002), a dynamic strat- egy in the error handling process is proposed. In all these studies, supervised learning techniques are used in order to classify dialogs to predict user satisfaction or dialog failures. A novel approach to the exploitation of dialog corpora is for speech recognition and language understanding modeling. In fact, such corpora allow for a multidimensional analysis of speech and language models of dialog systems. Our work differs from previous studies in the algo- rithmic approach and learning scenario. First we use unsupervised speech mining techniques. We apply data clustering methods to large spoken dialog corpora. Two kinds of clustering methods are used: a hierarchical one based on decision trees and the optimization of a statistical criterion; a partitional one based on a k-means algorithm applied to vectors rep- resenting the dialogs. We interpret the clusters obtained and define a label for each of them. Second we perform our analyses on large cor- pora of real dialogs collected from deployed sys- tems. These log files contain a trace of the …

Content truncated to 500 words