Effective Sequence-to-Sequence Dialogue State Tracking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7486–7493 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7486 Effective Sequence-to-Sequence Dialogue State Tracking Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, Yonghui Wu Google Research {jeffreyzhao,mahdis,yezhan,yuancao,yonghui}@google.com Abstract Sequence-to-sequence models have been ap- plied to a wide variety of NLP tasks, but how to properly use them for dialogue state tracking has not been systematically investi- gated. In this paper, we study this problem from the perspectives of pre-training objec- tives as well as the formats of context repre- sentations. We demonstrate that the choice of pre-training objective makes a significant difference to the state tracking quality. In particular, we find that masked span predic- tion is more effective than auto-regressive lan- guage modeling. We also explore using Pega- sus, a span prediction-based pre-training objec- tive for text summarization, for the state track- ing model. We found that pre-training for the seemingly distant summarization task works surprisingly well for dialogue state tracking. In addition, we found that while recurrent state context representation works also reasonably well, the model may have a hard time recov- ering from earlier mistakes. We conducted experiments on the MultiWOZ 2.1-2.4, WOZ 2.0, and DSTC2 datasets with consistent obser- vations. 1 Introduction Sequence-to-sequence (Seq2Seq) modeling (Sutskever et al., 2014) is one of the most widely adopted generative framework for a multitude of NLP tasks. While it has also been applied for task-oriented dialogue modeling (Wen et al., 2018; Lei et al., 2018; Rongali et al., 2020; Feng et al., 2020), how to best setup Seq2Seq models on this task remains an understudied topic. In this paper, we investigate this problem from two perspectives: Pre-training objectives and dialogue context representation, and we focus on the dialogue state tracking (DST) task. The flexibility of the Seq2Seq model allows us to adopt and compare pre-training strategies for other NLP tasks sharing the same architec- ture. Specifically, we first experimented with dif- ferent pre-training setups of T5 (Raffel et al., 2020) which have been shown to be effective for generic language understanding. Additionally, as an ex- ploratory effort, we applied Pegasus (Zhang et al., 2020b), a pre-training procedure designed for text summarization, to the task of DST. Additionally, to investigate how different dia- logue context representations affect Seq2Seq per- formance, we compare two versions of all models: one accepts full conversation history as context, and one that feeds the previously predicted states recurrently as summary of context. We conduct systematic experiments on the Multi- WOZ (Budzianowski et al., 2018) benchmark. For fair comparison with existing approaches, we re- port results on MultiWOZ 2.1-2.4 (Eric et al., 2019; Zang et al., 2020; Han et al., 2020; Ye et al., 2021), all 4 variations of the benchmark proposed to date. In addition, we report results on the WOZ 2.0 (Wen et al., 2016) and DSTC2 (Henderson et al., 2014) datasets. Our findings can be summarized as fol- lows: 1. Pre-training procedures involving masked span prediction work consistently better …

Content truncated to 500 words