FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6532–6546 July 9-14, 2023 ©2023 Association for Computational Linguistics FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue Weihao Zeng1⇤, Keqing He2⇤, Yejie Wang1, Chen Zeng1 Jingang Wang2, Yunsen Xian2, Weiran Xu1⇤ 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan, Beijing, China {zengwh,wangyejie,chenzeng,xuweiran}@bupt.edu.cn {hekeqing,wangjingang,xianyunsen}@meituan.com Abstract Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic pat- terns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice. Current dia- logue pre-training methods rely on a contrastive framework and face the challenges of both se- lecting true positives and hard negatives. In this paper, we propose a novel dialogue pre- training model, FutureTOD, which distills fu- ture knowledge to the representation of the previous dialogue context using a self-training framework. Our intuition is that a good dia- logue representation both learns local context information and predicts future information. Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, ro- bustness, and learning discriminative dialogue representations capabilities. 1 1 Introduction Pre-trained language models (Devlin et al., 2019; Liu et al., 2019) based on a massive scale of general text corpora (Zhu et al., 2015) have been commonly used in many NLP applications. Finetuning models on these PLMs significantly improves the perfor- mance of various downstream tasks, especially nat- ural language understanding. Despite their success, directly applying them to conversational corpora is proved to be suboptimal due to the large linguistic gap between conversations and plain text (Rashkin et al., 2019; Wolf et al., 2019). Therefore, it’s vital to explore dialogue-specific pre-trained models for solving various downstream dialogue tasks. Early pre-trained dialogue language models use chit-chat corpora from social media, such as Twit- ter or Reddit, aiming at retrieval (Henderson et al., ⇤The first two authors contribute equally. Weiran Xu is the corresponding author. 1Our code, models and other related resources are publicly available at https://github.com/Zeng-WH/FutureTOD Encoder Encoder Contrastive Loss Positive Response Negative Response Context Student Teacher Update Distillation Loss Context Context Future (a) Contrastive learning framework (CL) (b) No-contrastive self-training framework (FutureTOD) Figure 1: Comparison of different dialogue pre-training paradigms. The contrastive models learn context repre- sentations by pulling together positive pairs and pushing apart negative pairs. In contrast, our FutureTOD em- ploys a self-training framework to distill future knowl- edge to context representations and dismiss the require- ments of contrastive pairs. 2019) and dialogue response generation (Zhang et al., 2020). These open-domain dialogues are usually short, noisy, and without specific chatting goals. Further, a more practical scenario, task- oriented dialogue (TOD), is attracting more at- tention. TOD has explicit goals (e.g. restaurant reservation) and many conversational interactions like belief states and database information, making language understanding and policy learning more complex than those chit-chat scenarios. Each TOD dataset is usually small because collecting and la- beling such data are time-consuming. …

Content truncated to 500 words