Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7871–7881 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7871 Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking Weizhe Lin Bo-Hsiang Tseng Bill Byrne Department of Engineering, University of Cambridge, United Kingdom wl356@cam.ac.uk bht26@cam.ac.uk bill.byrne@eng.cam.ac.uk Abstract Dialogue State Tracking is central to multi- domain task-oriented dialogue systems, re- sponsible for extracting information from user utterances. We present a novel hybrid archi- tecture that augments GPT-2 with representa- tions derived from Graph Attention Networks in such a way to allow causal, sequential pre- diction of slot values. The model architecture captures inter-slot relationships and dependen- cies across domains that otherwise can be lost in sequential prediction. We report improve- ments in state tracking performance in Mul- tiWOZ 2.0 against a strong GPT-2 baseline and investigate a simplified sparse training sce- nario in which DST models are trained only on session-level annotations but evaluated at the turn level. We further report detailed anal- yses to demonstrate the effectiveness of graph models in DST by showing that the proposed graph modules capture inter-slot dependencies and improve the predictions of values that are common to multiple domains. 1 Introduction This paper investigates two aspects of dialogue state tracking (DST) for multi-domain task- oriented dialogue (Budzianowski et al., 2018). We present a novel hybrid architecture that augments GPT-2 (Radford et al., 2019) with dialogue act representations derived from Graph Attention Net- works (GATs) (Veliˇckovi´c et al., 2018) in such a way that allows causal, sequential prediction of slot values while explicitly modelling the relationships between slots and values across domains. Our ap- proach uses GATs to improve predictions of values that are shared across domain-slots and that might otherwise be treated independently. As a related line of work, we investigate a form of sparsely supervised DST training and find that our hybrid architecture offers improved robustness with weak supervision. DST can be improved by modelling the rela- tionship between slots and values across domains. This has been explored recently by Zhou and Small (2019) who suggest three types of relation- ships between domain-slots pairs that can be mod- elled explicitly: (1) pairs that share the same can- didate set, such as <restaurant-bookday> and <hotel-bookday>; (2) pairs whose can- didate values are subsets, as could happen with <restaurant-name> and <taxi-destina tion> if the candidate set of the first belongs to that of the second; and (3) correlated values be- tween domain-slot pairs, such as when the ‘star’ level of a booked hotel correlates with the price range of a reserved restaurant. Graph Neural Networks (GNNs) have been pro- posed to captures the interactions among slots and values and to improve DST performance (Zhou and Small, 2019; Chen et al., 2020; Wu et al., 2020). These relationships can be represented as edges in graph-based models, where domains, slots, and values are nodes in the graphs. However previous work has not explored quantitatively or in depth how graph models utilize the relationships they model. Chen et …

Content truncated to 500 words