Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10051 - 10063 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure Xueliang Zhao1,3∗, Lemao Liu2, Tingchen Fu4, Shuming Shi2, Dongyan Zhao1,3,5†, Rui Yan4† 1Wangxuan Institute of Computer Technology, Peking University 2Tencent AI Lab 3Center for Data Science, AAIS, Peking University 4Gaoling School of Artificial Intelligence, Renmin University of China 5Beijing Institute for General Artificial Intelligence {xl.zhao,zhaody}@pku.edu.cn {redmondliu,shumingshi}@tencent.com lucas.futingchen@gmail.com ruiyan@ruc.edu.cn Abstract With the availability of massive general- domain dialogue data, pre-trained dialogue gen- eration appears to be super appealing to transfer knowledge from the general domain to down- stream applications. In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability. This paper proposes a novel di- alogue generation model with a latent structure that is easily transferable from the general do- main to downstream tasks in a lightweight and transparent way. Experiments on two bench- marks validate the effectiveness of the proposed model. Thanks to the transferable latent struc- ture, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model with about 22% parameters particularly delivers a 5x speedup in running time compared with the strongest baseline. Moreover, the pro- posed model is explainable by interpreting the discrete latent variables. 1 Introduction Conversation between humans and machines has long been a goal of artificial intelligence (AI). Building an open-domain dialogue system with data-driven techniques has gotten a lot of attention in the AI and NLP fields in recent years, thanks to breakthroughs in deep learning (Sutskever et al., 2014; Gehring et al., 2017; Vaswani et al., 2017). In particular, with the availability of massive hu- man dialogue data (e.g., the Reddit comments) on social media (Adiwardana et al., 2020), pre-trained dialogue generation appears to be super appealing to alleviate potential discrepancies between general domain and downstream applications (Zhang et al., 2020; Bao et al., 2020, 2021; Li et al., 2021). *This work was done while X. Zhao was an intern at Ten- cent AI Lab. †Corresponding authors: Dongyan Zhao and Rui Yan. The common idea behind the pre-trained dia- logue generation can be highlighted as a two-step pipeline: a) it firstly trains a deep neural model on massive general-domain dialogue data, b) and then transfers the model into downstream tasks via fine- tuning or zero-shot learning. Under this pipeline, the transferability is mainly obtained by fitting a large model with millions of parameters on mas- sive data in an exhaustive way. Consequently, the downsides in existing works are obvious: their run- ning is inefficient and their outputs are difficult to explain. This paper thereby aims to build a pre-trained dialogue model which is easily transferable from the general domain to downstream tasks in a lightweight and transparent way. To …

Content truncated to 500 words