IBM’s Belief Tracker: Results On Dialog State Tracking Challenge Datasets

Proceedings of the of the EACL 2014 Workshop on Dialogue in Motion (DM), pages 10–18, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics IBM’s Belief Tracker: Results On Dialog State Tracking Challenge Datasets Rudolf Kadlec, Jindˇrich Libovick´y, Jan Macek, and Jan Kleindienst IBM Czech Republic V Parku 4, Prague 4 Czech Republic {rudolf kadlec, jindrich libovicky, jmacek2, jankle}@cz.ibm.com Abstract Accurate dialog state tracking is crucial for the design of an efficient spoken dialog system. Until recently, quantitative com- parison of different state tracking meth- ods was difficult. However the 2013 Dia- log State Tracking Challenge (DSTC) in- troduced a common dataset and metrics that allow to evaluate the performance of trackers on a standardized task. In this pa- per we present our belief tracker based on the Hidden Information State (HIS) model with an adjusted user model component. Further, we report the results of our tracker on test3 dataset from DSTC. Our tracker is competitive with trackers submitted to DSTC, even without training it achieves the best results in L2 metrics and it per- forms between second and third place in accuracy. After adjusting the tracker using the provided data it outperformed the other submissions also in accuracy and yet im- proved in L2. Additionally we present pre- liminary results on another two datasets, test1 and test2, used in the DSTC. Strong performance in L2 metric means that our tracker produces well calibrated hypothe- ses probabilities. 1 Introduction Spoken dialog systems need to keep a represen- tation of the dialog state and the user goal to follow an efficient interaction path. The perfor- mance of state-of-the-art speech recognition sys- tems varies widely with domain and environment with word accuracy rates ranging from less than 70% to 98%, which often leads to misinterpreta- tion of the user’s intention. Dialog state tracking methods need to cope with such error-prone auto- matic speech recognition (ASR) and spoken lan- guage understanding (SLU) outputs. Traditional dialog systems use hand-crafted rules to select from the SLU outputs based on their confidence scores. Recently, several data-driven approaches to dialog state tracking were developed as a part of end-to-end spoken dialog systems. However, specifics of these systems render comparison of dialog state tracking methods difficult. The Dialog State Tracking Challenge (DSTC) (Williams et al., 2013) provides a shared testbed with datasets and tools for evaluation of dialog state tracking methods. It abstracts from subsys- tems of end-to-end spoken dialog systems focus- ing only on the dialog state estimation and track- ing. It does so by providing datasets of ASR and SLU outputs with reference transcriptions together with annotation on the level of dialog acts. In this paper we report initial encouraging re- sults of our generative belief state tracker. We plan to investigate discriminative approaches in the fu- ture. The rest of the paper continues as follows. In the next section we formally introduce the dia- log tracking task together with datasets used in the DSTC. Then in Section 3 we discuss related work. Section 4 …

Content truncated to 500 words