Learning to Customize Model Structures for Few-shot Dialogue Generation Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5832–5841 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5832 Learning to Customize Model Structures for Few-shot Dialogue Generation Tasks Yiping Song1, Zequn Liu1, Wei Bi2, Rui Yan3, Ming Zhang1∗ 1Department of Computer Science, School of EECS, Peking University, Beijing, China 2Tencent AI Lab, Shenzhen, China 3Wangxuan Institute of Computer Technology, Peking University, Beijing, China songyiping,zequnliu,mzhang cs,ruiyan@pku.edu.cn victoriabi@tencent.com Abstract Training the generative models with minimal corpus is one of the critical challenges for building open-domain dialogue systems. Ex- isting methods tend to use the meta-learning framework which pre-trains the parameters on all non-target tasks then fine-tunes on the tar- get task. However, fine-tuning distinguishes tasks from the parameter perspective but ig- nores the model-structure perspective, result- ing in similar dialogue models for different tasks. In this paper, we propose an algorithm that can customize a unique dialogue model for each task in the few-shot setting. In our approach, each dialogue model consists of a shared module, a gating module, and a pri- vate module. The first two modules are shared among all the tasks, while the third one will differentiate into different network structures to better capture the characteristics of the cor- responding task. The extensive experiments on two datasets show that our method outper- forms all the baselines in terms of task consis- tency, response quality, and diversity. 1 Introduction Generative dialogue models often require a large amount of dialogues for training, and it is chal- lenging to build models that can adapt to new domains or tasks with limited data. With recent advances in large-scale pre-training [Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018], we can first pre-train a generative model on large-scale dialogues from the non-target domains and then fine-tune on the task-specific data corpus [Wang et al., 2019a; Alt et al., 2019a; Klein, 2019]. While pre-training is beneficial, such models still require sufficient task- specific data for fine-tuning. They cannot achieve satisfying performance when very few examples ∗Corresponding author are given [Bansal et al., 2019]. Unfortunately, this is often the case in many dialogue generation sce- narios. For example, in personalized dialogue gen- eration, we need to quickly adapt to the response style of a user’s persona by just a few his or her di- alogues [Madotto et al., 2019; Zhang et al., 2018]; in emotional dialogue generation, we need to gen- erate a response catering to a new emoji using very few utterances containing this emoji [Zhou et al., 18; Zhou and Wang, 2018]. Hence, this is the fo- cus of our paper - few-shot dialogue generation, i.e. training a generative model that can be gener- alized to a new task (domain) within k-shots of its dialogues. A few works have been proposed to consider few-shot dialogue generation as a meta-learning problem [Madotto et al., 2019; Qian and Yu, 2019; Mi et al., 2019]. They all rely on the popular model- agnostic …

Content truncated to 500 words