Pretrained Language Models for Dialogue Generation with Multiple Input Sources

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 909–917 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 909 Pretrained Language Models for Dialogue Generation with Multiple Input Sources Yu Cao 1∗, Wei Bi 2, Meng Fang 3, Dacheng Tao 1 1 UBTECH Sydney AI Center, School of Computer Science, Faculty of Engineering, The University of Sydney, Australia 2 Tencent AI LAB, China, 3 Tencent Robotics X, China ycao8647@uni.sydney.edu.au, victoriabi@tencent.com, mfang@tencent.com, dacheng.tao@sydney.edu.au Abstract Large-scale pretrained language models have achieved outstanding performance on natu- ral language understanding tasks. However, it is still under investigating how to apply them to dialogue generation tasks, especially those with responses conditioned on multiple sources. Previous work simply concatenates all input sources or averages information from different input sources. In this work, we study dialogue models with multiple input sources adapted from the pretrained language model GPT2. We explore various methods to fuse multiple separate attention information corre- sponding to different sources. Our experimen- tal results show that proper fusion methods deliver higher relevance with dialogue history than simple fusion baselines. 1 Introduction Large-scale pretrained language models (Devlin et al., 2019; Radford et al., 2018, 2019) have achieved outstanding performance on various natural language understanding tasks (Young et al., 2018; Liu et al., 2019). Researchers have then utilized them in dialogue generation tasks (Budzianowski and Vuli´c, 2019; Edunov et al., 2019; Zhang et al., 2019). Many of them sim- ply concatenate the input dialogue history and the output response in finetuning, since the pretrained language model only accepts a single sequence as input. However, dialogue generation tasks may in- volve multiple input sources simultaneously. For example, in personalized or knowledge-grounded dialogue generation (Li et al., 2016; Zhang et al., 2018; Dinan et al., 2018), a response is generated conditioned on both dialogue history and an aux- iliary user profile or knowledge article. Despite ∗This work was done during Yu Cao’s internship in Ten- cent AI LAB, Shenzhen. simple concatenation of all input sources, an impor- tant question arises on how we can better adapt a single-input pretrained language model to a multi- input dialogue generation task. Some previous work forms an encoder-decoder architecture with both encoder and decoder du- plicated from a pretrained language model (Golo- vanov et al., 2019; Zheng et al., 2019). Recently, BART (Lewis et al., 2019) even obtain a complete pretrained model under this architecture directly. Taking personalized dialogue generation (Zhang et al., 2018) as an example, we can treat persona in- formation, dialogue history and previous generated tokens as three different input sources. The for- mer two will be encoded firstly and then combined with the last one in the decoder. In Golovanov et al. 2019, the multi-head attention layer in the decoder is copied three times for each input source and mean pooling is used to average results from multiple attentions. This encoder-decoder adapta- tion is shown to outperform simple concatenation. However, when dialogue history gets longer, this model tends to use less information of each dia- …

Content truncated to 500 words