Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1096 - 1106 December 7-11, 2022 ©2022 Association for Computational Linguistics Curriculum Prompt Learning with Self-Training for Abstractive Dialogue Summarization Changqun Li1, Linlin Wang1∗, Xin Lin1, Gerard de Melo2, Liang He1 1 East China Normal University 2 Hasso Plattner Institute / University of Potsdam 52215901009@stu.ecnu.edu.cn, {llwang,xlin,lhe}@cs.ecnu.edu.cn, gdm@demelo.org Abstract Succinctly summarizing dialogue is a task of growing interest, but inherent challenges, such as insufficient training data and low informa- tion density impede our ability to train abstrac- tive models. In this work, we propose a novel curriculum-based prompt learning method with self-training to address these problems. Specif- ically, prompts are learned using a curriculum learning strategy that gradually increases the degree of prompt perturbation, thereby improv- ing the dialogue understanding and modeling capabilities of our model. Unlabeled dialogue is incorporated by means of self-training so as to reduce the dependency on labeled data. We further investigate topic-aware prompts to better plan for the generation of summaries. Ex- periments confirm that our model substantially outperforms strong baselines and achieves new state-of-the-art results on the AMI and ICSI datasets. Human evaluations also show the su- periority of our model with regard to the sum- mary generation quality. 1 Introduction As billions of people engage in instant messag- ing and other forms of interaction, there is no- table interest in techniques to process and dis- till recorded dialogues into concise and natural summaries (Gurevych and Strube, 2004; Murray et al., 2006). The inherent challenges of the di- alogue make abstractive dialogue summarization particularly difficult. First, the available labeled data is substantially smaller than news summa- rization data, e.g., 137 meetings in AMI (Mc- Cowan et al., 2005) vs. 312K articles in CNN/Daily Mail (Hermann et al., 2015). Second, everyday di- alogue involves a dynamic information exchange flow (Sacks et al., 1978) such that salient informa- tion is often scattered across multiple utterances by different interlocutors (Li and Choi, 2020). ∗ Corresponding author. Email: llwang@cs.ecnu.edu.cn. The first two authors contributed equally to this work. Dialogue That’s good, running a company. C3 C2 C1 No, I sold off the office. Victor, I heard you took over Chris’s company. ... Yes. Chris was under huge debt. ... David, how is your business? It’s good, but no work these days. Yes, market is very slow. Summary: Victor took over Chris’s company. It was under a huge debt, and he sold the office. David’s business goes slow. s1 s2 s3 Figure 1: A dialogue and its summary, consisting of three sentences s1, s2, s3. For illustration purposes, we show relevant content snippets C1, C2, C3 in the dia- logue that these correspond to, and have also underlined salient information that appears in the summary. Recently, there is a growing trend of using Pre- trained Language Models (PLMs) for dialogue summarization (Gliwa et al., 2019; Fabbri et al., 2021). However, these models tend to require abun- dant labeled dialogue, which is difficult to pro- cure for low-resource …

Content truncated to 500 words