Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking

Findings of the Association for Computational Linguistics: ACL 2022, pages 3824 - 3846 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking Jamin Shin1∗, Hangyeol Yu1∗, Hyeongdon Moon1∗, Andrea Madotto2, Juneyoung Park1 1Riiid AI Research 2The Hong Kong University of Science and Technology jayshin.nlp@gmail.com, {hangyeol.yu,hyeongdon.moon}@riiid.co amadotto@connect.ust.hk, juneyoung.park@riiid.co Abstract Annotating task-oriented dialogues is notori- ous for the expensive and difficult data collec- tion process. Few-shot dialogue state tracking (DST) is a realistic solution to this problem. In this paper, we hypothesize that dialogue sum- maries are essentially unstructured dialogue states; hence, we propose to reformulate di- alogue state tracking as a dialogue summariza- tion problem. To elaborate, we train a text-to- text language model with synthetic template- based dialogue summaries, generated by a set of rules. Then, the dialogue states can be recov- ered by inversely applying the summary gen- eration rules. We empirically show that our method DS2 outperforms previous works on few-shot DST in MultiWoZ 2.0 and 2.1, in both cross-domain and multi-domain settings. Our method1 also exhibits vast speedup during both training and inference as it can generate all states at once. Finally, based on our analysis, we discover that the naturalness of the sum- mary templates plays a key role for successful training. 1 Introduction Task-oriented dialogue systems (TOD) have pene- trated our daily lives much more than before and they will continue to increase their presence. For example, many of our mobile devices are equipped with such dialogue agents like Siri, and we now often encounter customer service or flight reser- vation bots. Dialogue State Tracking (DST) is an essential element of such task-oriented dialogue systems (Wu et al., 2019; Balaraman et al., 2021). The main goal of this component is to understand the user’s requirements expressed during the con- versation under a given schema or ontology. Hence, ∗ Equal Contribution: JS proposed the main idea and scaled up the experiments. HY designed and implemented the heuristic state tracking component. HM conducted rapid prototyping, analysis, and ablations. 1Code: github.com/jshin49/ds2 Figure 1: Example dialogue in taxi domain, its dialogue state, and template summary created from the state. as shown in Figure 1, accurately extracting the de- parture, destination, and arrival time of the user is key to creating a good user experience. However, collecting such turn-level dialogue state annotations is very expensive and requires a significant amount of design and mediating ef- forts from domain experts (Budzianowski et al., 2018; Eric et al., 2020; Park et al., 2021). This is because the collection process follows the Wizard- of-Oz (WoZ) style (Kelley, 1984), which requires two human workers to converse with each other and annotate the states for each turn. To cope with this inherent scalability issue, Budzianowski et al. (2018) attempted to crowd-source this process in MultiWoZ 2.0 which resulted in one of the largest publicly available multi-domain task-oriented di- alogue dataset. However, the resulting dataset is very noisy in data annotations which often hindered …

Content truncated to 500 words