BRIO: Bringing Order to Abstractive Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2890 - 2903 May 22-27, 2022 c⃝2022 Association for Computational Linguistics BRIO: Bringing Order to Abstractive Summarization Yixin Liu1, Pengfei Liu2, Dragomir Radev1, Graham Neubig2 1Yale University, 2Carnegie Mellon University {yixin.liu,dragomir.radev}@yale.edu,{pliu3,gneubig}@cs.cmu.edu Abstract Abstractive summarization models are com- monly trained using maximum likelihood es- timation, which assumes a deterministic (one- point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during infer- ence, where the model needs to compare sev- eral system-generated (candidate) summaries that have deviated from the reference sum- mary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate proba- bilities of candidate summaries that are more correlated with their level of quality.1 1 Introduction Neural methods for abstractive summariza- tion (Rush et al., 2015; Nallapati et al., 2016; Chopra et al., 2016; Lewis et al., 2020; Zhang et al., 2020) formulate summarization as a sequence- to-sequence (Seq2Seq) problem (Sutskever et al., 2014), learning to generate the summary in an autoregressive manner. Such models are com- monly trained with maximum likelihood estima- tion (MLE), maximizing predictive probability of the reference output given the gold sub-sequence before it. However, during inference the model must also generate the output based on possibly erroneous previous steps. This can hurt model per- formance, a phenomenon often called exposure bias (Bengio et al., 2015; Ranzato et al., 2016). To maintain reasonable performance even in the case of a sub-sequence with errors, we argue that the 1We have made our code, results, and trained models pub- licly available at https://github.com/yixinL7/BRIO. System R-1 R-2 R-L Acc.(%) High 53.99 29.85 51.12 100.00 Low 33.48 10.85 30.45 0.00 BART 44.88 21.68 41.92 54.80 Ours 50.10 26.29 47.19 79.63 Table 1: Accuracy of different abstractive summarization systems w.r.t ranking the quality of candidate summaries on CNNDM dataset. Acc. stands for the frequency of the model assigning higher probabilities to better candidate summaries. The candidate summaries are generated by a pre-trained model (BART), and we select the best and the worst candidates (w.r.t. ROUGE scores) for each of the samples. High and Low repre- sent the average performance of the best and worst candidates respectively. R-1/2/L are the ROUGE-1/2/L scores. The origi- nal BART only achieves 54.80% accuracy. model must accurately estimate relative quality of different generated outputs, since effective infer- ence requires comparison among these candidates. To understand whether existing models can ac- curately perform such relative comparisons, we conducted a preliminary study on pre-trained BART (Lewis et al., 2020), first generating two candidate summaries from the model and observ- ing whether a higher probability is assigned to the candidate with a …

Content truncated to 500 words