Evaluating language understanding accuracy with respect to objective outcomes in a dialogue system

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 471–481, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics Evaluating language understanding accuracy with respect to objective outcomes in a dialogue system Myroslava O. Dzikovska and Peter Bell and Amy Isard and Johanna D. Moore Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh, United Kingdom {m.dzikovska,peter.bell,amy.isard,j.moore}@ed.ac.uk Abstract It is not always clear how the differences in intrinsic evaluation metrics for a parser or classifier will affect the performance of the system that uses it. We investigate the relationship between the intrinsic evalua- tion scores of an interpretation component in a tutorial dialogue system and the learn- ing outcomes in an experiment with human users. Following the PARADISE method- ology, we use multiple linear regression to build predictive models of learning gain, an important objective outcome metric in tutorial dialogue. We show that standard intrinsic metrics such as F-score alone do not predict the outcomes well. However, we can build predictive performance func- tions that account for up to 50% of the vari- ance in learning gain by combining fea- tures based on standard evaluation scores and on the confusion matrix entries. We argue that building such predictive mod- els can help us better evaluate performance of NLP components that cannot be distin- guished based on F-score alone, and illus- trate our approach by comparing the cur- rent interpretation component in the system to a new classifier trained on the evaluation data. 1 Introduction Much of the work in natural language processing relies on intrinsic evaluation: computing standard evaluation metrics such as precision, recall and F- score on the same data set to compare the perfor- mance of different approaches to the same NLP problem. However, once a component, such as a parser, is included in a larger system, it is not always clear that improvements in intrinsic eval- uation scores will translate into improved over- all system performance. Therefore, extrinsic or task-based evaluation can be used to complement intrinsic evaluations. For example, NLP com- ponents such as parsers and co-reference resolu- tion algorithms could be compared in terms of how much they contribute to the performance of a textual entailment (RTE) system (Sammons et al., 2010; Yuret et al., 2010); parser performance could be evaluated by how well it contributes to an information retrieval task (Miyao et al., 2008). However, task-based evaluation can be difficult and expensive for interactive applications. Specif- ically, task-based evaluation for dialogue systems typically involves collecting data from a number of people interacting with the system, which is time-consuming and labor-intensive. Thus, it is desirable to develop an off-line evaluation pro- cedure that relates intrinsic evaluation metrics to predicted interaction outcomes, reducing the need to conduct experiments with human participants. This problem can be addressed via the use of the PARADISE evaluation methodology for spo- ken dialogue systems (Walker et al., 2000). In a PARADISE study, after an initial data collection with …

Content truncated to 500 words