What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10520–10542 July 9-14, 2023 ©2023 Association for Computational Linguistics What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization Griffin Adams♠,♣∗ griffin.adams@columbia.edu Bichlien H Nguyen♢ bnguy@microsoft.com Jake Smith♢ jakesmith@microsoft.com Yingce Xia♢ yingce.xia@microsoft.com Shufang Xie♢ shufxi@microsoft.com Anna Ostropolets♣ anna.ostropolets@columbia.edu Budhaditya Deb♢ budha.deb@microsoft.com Yuan-Jyue Chen♢ yuanjc@microsoft.com Tristan Naumann♢ tristan@microsoft.com Noémie Elhadad♠,♣ noemie.elhadad@columbia.edu Microsoft Research♢ Columbia University: Computer Science♠, Biomedical Informatics♣ Abstract Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE). To address this, re- cent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness. While effective, much of this work has focused on how to generate and optimize these sets. Less is known about why one setup is more effective than another. In this work, we uncover the underlying characteristics of effective sets. For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for cali- bration fine-tuning. Each selection strategy tar- gets distinct aspects of the sets, such as lexical diversity or the size of the gap between posi- tive and negatives. On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibra- tion is optimal when the negative sets are extrac- tive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise–the disagreement between model and metric defined candidate rankings–minimized. Code to create, select, and optimize calibration sets is available at https://github.com/ griff4692/calibrating-summaries. ∗Work started during internship with Microsoft Research. 1 Introduction Traditionally, summarization models have been trained to maximize the likelihood of gold-standard references. This training paradigm introduces an exposure bias because, during training, the model is not exposed to the metrics on which it is evaluated. Without being able to calibrate its own predictions with metrics, models are prone to produce sum- maries with irrelevant or repetitive content (Zhao et al., 2022), or misrepresent the claims in the source text (Cao et al., 2018; Maynez et al., 2020). Calibration offers a flexible and effective set of methods to remedy this exposure bias by explicitly instructing a model to distinguish between high and low quality summaries. By varying how can- didate sets are constructed and optimized, an extra calibration step can unlock large gains in relevance (via ROUGE (Liu and Liu, 2021a; Liu et al., 2022)) or improve the faithfulness of summaries to the source (Nan et al., 2021b; Cao and Wang, 2021a). Yet, much of this work has addressed how— how to generate candidates (Cao and Wang, 2021a) and how to define effective calibration objectives (Nan et al., 2021b; Zhao et al., 2022). Work has largely been separated into relevance …

Content truncated to 500 words