DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4852 - 4864 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation Wei Chen1∗, Yeyun Gong2, Song Wang3, Bolun Yao4, Weizhen Qi2, Zhongyu Wei1, Xiaowu Hu2, Bartuer Zhou2, Yi Mao3, Weizhu Chen3, Biao Cheng2 and Nan Duan2 1School of Data Science, Fudan University, China 2Microsoft Research Asia, China 3Microsoft, US 4Nanjing University of Science and Technology, China {chenwei18,zywei}@fudan.edu.cn; {yegong,weizhen,xiaowuhu,bazhou,bicheng,nanduan}@microsoft.com; {sonwang,yimao,wzchen}@microsoft.com; yaobl001@njust.edu.cn Abstract Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses. In this paper, we propose a new dia- log pre-training framework called DialogVED, which introduces continuous latent variables into the enhanced encoder-decoder pre-training framework to increase the relevance and di- versity of responses. With the help of a large dialog corpus (Reddit), we pre-train the model using the following 4 tasks, used in training lan- guage models (LMs) and Variational Autoen- coders (VAEs) literature: 1) masked language model; 2) response generation; 3) bag-of-words prediction; and 4) KL divergence reduction. We also add additional parameters to model the turn structure in dialogs to improve the perfor- mance of the pre-trained model. We conduct experiments on PersonaChat, DailyDialog, and DSTC7-AVSD benchmarks for response gen- eration. Experimental results show that our model achieves the new state-of-the-art results on all these datasets. 1 Introduction Pre-trained language models (PLMs) have been widely explored both in natural language under- standing (NLU) and generation (NLG) in recent years, this pre-training and fine-tuning paradigm sheds light on various downstream tasks in natural language processing (NLP). Compared with gen- eral pre-trained models, task-oriented pre-trained models (such as Summarization, Dialog and etc.), which is designed in line with task characteristics, may achieve better performance and be more ro- bust. In this paper, we proposes a novel pre-trained dialog response generation model based on previ- ous research. Dialogue Response Generation (DSG) in open domain is a challenging task with a wide range of ∗Worked during the internship at Microsoft Research Asia. Zhongyu Wei and Yeyun Gong are corresponding authors. application scenarios. Recent advances in DSG utilize pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and GPT2 (Radford et al., 2019) in two major categories. The first one focuses on how to fine-tune PLMs in downstream tasks and address the various application-specific needs and challenges (Lin et al., 2020). The sec- ond one augments dialog specific tasks into the PLM training (Zhang et al., 2020; Bao et al., 2020) and then fine-tunes the new pre-trained model in downstream tasks. We study the latter in this paper. There is a proverbial one-to-many problem in DSG, i.e., a single dialog context could be followed by multiple reasonable responses. Existing works introduce latent variables to model this problem. For example, VHRED (Serban et al., 2017) incorpo- rates latent continuous variable into the sequence- to-sequence (Seq2Seq) RNN model to improve the diversity …

Content truncated to 500 words