BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2832–2846 July 9-14, 2023 ©2023 Association for Computational Linguistics BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking Seunpgil Won1,2 Heeyoung Kwak4,5 Joongbo Shin1 Janghoon Han1 Kyomin Jung2,3 1LG AI Research, 2Seoul National University, 3SNU-LG AI Research Center 4NAVER AI Lab, 5NAVER Digital Healthcare Lab {seungpil.won, jb.shin, janghoon.han}@lgresearch.ai heeyoung.kwak@navercorp.com kjung@snu.ac.kr Abstract Despite the recent advances in dialogue state tracking (DST), the joint goal accuracy (JGA) of the existing methods on MultiWOZ 2.1 still remains merely 60%. In our preliminary error analysis, we find that beam search produces a pool of candidates that is likely to include the correct dialogue state. Motivated by this obser- vation, we introduce a novel framework, called BREAK (Beam search and RE-rAnKing), that achieves outstanding performance on DST. Our proposed method performs DST in two stages: (i) generating k-best dialogue state candidates with beam search and (ii) re-ranking the candidates to select the correct dialogue state. This simple yet powerful framework shows state-of-the-art performance on all versions of MultiWOZ and M2M datasets. Most notably, we push the joint goal accuracy to 80-90% on MultiWOZ 2.1-2.4, which is an improvement of 23.6%, 26.3%, 21.7%, and 10.8% over the previous best-performing models, respec- tively. The data and code will be available at https://github.com/tony-won/DST -BREAK. 1 Introduction Dialogue state tracking (DST) is an essential component of task-oriented dialogue (TOD) sys- tems to help users achieve their specific goals, such as booking restaurants or finding attrac- tions (Budzianowski et al., 2018). The task of DST is to understand the meaning of user utterances and keep track of users’ intentions throughout the conversation. Since the results of DST affects the subsequent TOD tasks, i.e., dialogue policy and re- sponse generation, the accuracy of DST is crucial without a doubt (Kim et al., 2020; Lee et al., 2019). In DST, the dialogue state is typically represented by a set of (slot, value) pairs, e.g., (“hotel-area”, “centre”). Here, the list of slots is a pre-defined set, and the corresponding values are extracted from the dialogue context. Generation-based DST with Greedy Search Decoder Rank Word Prob. 1 monday 76.8% 2 sunday 21.4% 3 none 1.2% ⁞ 32k restaurant-book day User : I’m looking for an Italian restaurant. Sys : How about Duomo? Would you like a reservation? User : Yes, book me a table for 2 people on . Let’s say 20:00? Dialogue Context Sunday Softmax restaurant-food : italian restaurant-name : duomo restaurant-book day : restaurant-book people : 2 restaurant-book time : 20:00 monday Dialogue State Prediction Error Analysis Figure 1: An example of dialogue state tracking with a generation-based model and its failure case. Greedy search fails to generate the accurate slot value for restaurant-book day. However, the output probability of the correct value sunday still ranks very high, provid- ing a rationale for using beam search to reconsider the high-ranking tokens. Thanks to large-scale pre-trained language models (PLMs) (Devlin et al., 2019; …

Content truncated to 500 words