Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9649 - 9668 December 7-11, 2022 ©2022 Association for Computational Linguistics REFEREE: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation Melanie Sclar1 Peter West1 Sachin Kumar2 Yulia Tsvetkov1 Yejin Choi1,3 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Language Technologies Institute, Carnegie Mellon University 3Allen Institute for Artificial Intelligence msclar@cs.washington.edu Abstract We present REFEREE, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold sum- maries for supervision), while allowing direct control for compression ratio. Our work is the first to demonstrate that reference-free, con- trolled sentence summarization is feasible via the conceptual framework of Symbolic Knowl- edge Distillation (West et al., 2022), where la- tent knowledge in pre-trained language models is distilled via explicit examples sampled from the teacher models, further purified with three types of filters: length, fidelity, and Informa- tion Bottleneck. Moreover, we uniquely pro- pose iterative distillation of knowledge, where student models from the previous iteration of distillation serve as teacher models in the next iteration. Starting off from a relatively modest set of GPT3-generated summaries, we demon- strate how iterative knowledge distillation can lead to considerably smaller, but better sum- marizers with sharper controllability. A useful by-product of this iterative distillation process is a high-quality dataset of sentence-summary pairs with varying degrees of compression ra- tios. Empirical results demonstrate that the fi- nal student models vastly outperform the much larger GPT3-Instruct model in terms of the con- trollability of compression ratios, without com- promising the quality of resulting summariza- tion.1 1 Introduction We introduce REFEREE, a new framework for sen- tence summarization that works by iteratively gen- erating and distilling knowledge into successively better models. This allows REFEREE to be [Re- fer]ence fr[ee]—beginning by distilling from a large language model rather than with supervised data. Yet, our method results in a more efficient, 1See https://github.com/msclar/referee for code, models, and data. ! ! REFEREE DISTILL ! ! control codes NLI Length IB REFEREE CONTROL GPT-3 ! Figure 1: Our method results in high quality, reference- free compact summarizers. We begin by using a large language model (e.g. GPT-3) to generate many summaries that demonstrate different aspects we may want in a summary—gray represents an aspect well- represented in these generations, while black is under- represented. We first use REFEREE-DISTILL to itera- tively filter and train summarizers that better represent these desirable aspects, e.g. shorter summary length. We then use generations from REFEREE-DISTILL to train a model in which these aspects are controllable: this is REFEREE-CONTROL. compact, and controllable summarization model than what we start with. Our work follows the paradigm of Symbolic Knowledge Distillation (West et al., 2022), which transfers implicit knowledge from a massive lan- guage model to a considerably smaller student model by explicitly generating knowledge in tex- tual form. Unlike traditional knowledge distillation (Hinton et al., 2015) where the teacher model and the student model are of the same …

Content truncated to 500 words