Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12737–12755 July 9-14, 2023 ©2023 Association for Computational Linguistics SOCRATIC Pretraining: Question-Driven Pretraining for Controllable Summarization Artidoro Pagnoni∗1 Alexander R. Fabbri2 Wojciech Kry´sci´nski2 Chien-Sheng Wu2 1University of Washington, 2Salesforce AI Research artidoro@uw.edu, {afabbri, wojciech.kryscinski, wu.jason}@salesforce.com Abstract In long document controllable summarization, where labeled data is scarce, pretrained mod- els struggle to adapt to the task and effectively respond to user queries. In this paper, we intro- duce SOCRATIC pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summa- rization tasks. By training a model to generate and answer relevant questions in a given con- text, SOCRATIC pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be sum- marized. We demonstrate the effectiveness of this approach through extensive experimenta- tion on two summarization domains, short sto- ries and dialogue, and multiple control strate- gies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question genera- tion system and outperforms pre-finetuning ap- proaches that use additional supervised data. Furthermore, our results show that SOCRATIC pretraining cuts task-specific labeled data re- quirements in half, is more faithful to user- provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY. 1 Introduction Summarization systems are designed to help users navigate large amounts of information (Edmunds and Morris, 2000), but often fail to meet the unique needs of different users, especially for long docu- ments. Recent research has explored ways to make summarization systems more controllable (Born- stein et al., 1999; Leuski et al., 2003) by allowing users to input queries or control sequences such as keywords (He et al., 2020), questions (Zhong et al., 2021), entity chains (Narayan et al., 2021), or question-answer pairs (Narayan et al., 2022). A challenge shared by all of the mentioned ap- proaches is the absence of abundant labeled data. ∗ Work done during internship at Salesforce In a group discussion about a philosophical concept, Sarah used the Socratic method by asking and answering questions to stimulate critical thinking and clarify underlying assumptions. The method helped her and her classmates achieve a deeper understanding of the concept and address areas of disagreement. Target: Document In a group [Mask] philosophical concept, Sarah [Mask] method by asking [Mask] questions to stimulate critical thinking [Mask] clarify [Mask]. The method helped her and [Mask] understanding of the concept and address areas of disagreement. Input: Masked Document Denoising Pretraining How did Sarah use the Socratic method in a group discussion? [QSep] In a group discussion about a philosophical concept, Sarah used the Socratic method by asking and answering questions to stimulate critical thinking and clarify underlying assumptions. Target: Generated Question + Pseudo-Summary [Ask&Answer] [Mask] The method helped her and her classmates achieve a deeper understanding of the concept and address areas of disagreement. Input: Masked Document Socratic Pretraining Figure 1: Our SOCRATIC pretraining compared to de- noising. We mask important …

Content truncated to 500 words