Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4848 - 4862 July 10-15, 2022 ©2022 Association for Computational Linguistics Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization Lulu Zhao1∗, Fujia Zheng1∗, Weihao Zeng1, Keqing He2, Weiran Xu1†, Huixing Jiang2, Wei Wu2, Yanan Wu1 1Pattern Recognition & Intelligent System Laboratory 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan Group, Beijing, China {zhaoll,fujia_zheng,ZengWH,xuweiran}@bupt.edu.cn {kqin}@bupt.cn, {jhx_bupt}@163.com, {wuwei19850318}@gmail.com Abstract The most advanced abstractive dialogue sum- marizers lack generalization ability on new domains and the existing researches for do- main adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and general- izable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot ex- periments and build domain adaptation bench- marks on two multi-domain dialogue summa- rization datasets, TODSum and QMSum. Ad- equate experiments and qualitative analysis prove the effectiveness of our methods. 1 Introduction Abstractive dialogue summarization task aims to distill the most critical information in a conversa- tion to produce a concise version, involving chit- chat (Gliwa et al., 2019; Chen and Yang, 2020), meeting(Zhong et al., 2021), customer service (Liu et al., 2019; Zou et al., 2021b), and task-oriented dialogue scenarios (Zhao et al., 2021b). Compared to the single-speaker texts, summarizing a dialogue presents a unique set of challenges, such as un- structured expressions and information sparsity is- sues. Recently, large-scale generative pre-trained models (Lewis et al., 2020; Liu and Lapata, 2019) have promoted the development of abstractive di- alogue summarization but they all require exten- sive human-annotated golden summaries, which makes them not scalable to new domains where only few/no labeled data is available. Considering that real-world applications often face the problem ∗ The first two authors contribute equally. † Weiran Xu is the corresponding author. U: I am looking for a train. The train should arrive by 9:45 and should depart from Cambridge. …… A: What is your destination? U: I need to go to Norwich. A: How about TR2534 that leaves at 5:36? …… Professor A: if you found that nine was better than six that would be OK. …… Professor A: Yeah, just work with the models, yeah. PhD B: yeah, just look at the length of the models and just see what happens. …… Inform (destination=Norwich; arrive by=9:45; departure=Cambridge) What did the professor think about improving the back recognizer for the Aurora task? The user asks duration, arrive by and train id. The departure of the t rai n is Cam bri dge an d th e destination is Norwich, and the train arrives at 9:45 on Friday. The Professor thought that the experiments were pretty simple, so computational power was not really a problem. He also …

Content truncated to 500 words