Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3096–3102 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3096 Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles Carel van Niekerk, Michael Heck, Christian Geishauser Hsien-Chin Lin, Nurul Lubis, Marco Moresi, Milica Gaˇsi´c Heinrich Heine University D¨usseldorf, Germany niekerk,heckmi,geishaus,linh,lubis,moresi,gasic@hhu.de Abstract The ability to accurately track what happens during a conversation is essential for the per- formance of a dialogue system. Current state- of-the-art multi-domain dialogue state trackers achieve just over 55% accuracy on the current go-to benchmark, which means that in almost every second dialogue turn they place full con- fidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a dis- tribution over possible dialogue states. How- ever, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibra- tion for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outper- forms previous dialogue belief tracking mod- els in terms of accuracy. 1 Introduction Task-oriented dialogue systems aim to act as assis- tants to their users, solving tasks such as finding a restaurant, booking a train, or providing informa- tion about a tourist attraction. They have become very popular with the introduction of virtual assis- tants such as Siri and Alexa. Two tasks are fundamental to such a system. The first is the ability to track what happened in the conversation, referred to as tracking. Based on the result of tracking, the system needs to conduct the conversation towards the fulfilment of the user goal, referred to as planning. The tracking component summarises the dialogue history, or the past, while the planning component manages the dialogue and concerns the future. In this work we focus on the first component. Early approaches to statistical dialogue mod- elling view dialogue as a Markov decision pro- cess (Levin et al., 1998) and define a set of dia- logue states that the conversation can be in at any given dialogue turn. The tracking component tracks the dialogue state. In recent years discrimina- tive models achieve state-of-the-art dialogue state tracking (DST) results (Kim et al., 2019; Zhang et al., 2019; Heck et al., 2020). Still, in a multi- domain setting such as MultiWOZ (Eric et al., 2019; Budzianowski et al., 2018), they achieve an accuracy of just over 55%. This means that in approximately 45% of cases they make a wrong prediction and, even worse, they have full confi- dence in that wrong prediction. In the wake of statistical dialogue modeling, the use of partially observable Markov decision pro- cesses has been proposed to address this issue. The idea is to model the probability over all possible dialogue states in every dialogue turn (Williams and Young, 2007). This probability distribution is referred to as the belief state. The advantages of belief tracking are probably best illustrated by an excerpt from a dialogue with a real …

Content truncated to 500 words