SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4504 - 4524 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization Mathieu Ravaut♣♢, Shafiq Joty∗♣♠, Nancy F. Chen∗♢ ♣ Nanyang Technological University, Singapore ♢ Institute of Infocomm Research (I2R), Singapore ♠ Salesforce Research Asia, Singapore {mathieuj001@e.ntu, srjoty@ntu}.edu.sg nfychen@i2r.a-star.edu.sg Abstract Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the down- stream dataset. These models are typically de- coded with beam search to generate a unique summary. However, the search space is very large, and with the exposure bias, such decod- ing is not optimal. In this paper, we show that it is possible to directly train a second- stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candi- date and consistently improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN- DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the- art. Our code and checkpoints will be avail- able at https://github.com/ntunlp/ SummaReranker. 1 Introduction In recent years, sequence-to-sequence neural mod- els have enabled great progress in abstractive sum- marization (See et al., 2017; Lin et al., 2021). In the news domain, they have surpassed the strong LEAD-3 extractive baseline. With the rise of transfer learning since BERT (Devlin et al., 2019), leading approaches typically fine-tune a base pre- trained model that either follows a general text generation training objective like T5 (Raffel et al., 2019), BART (Lewis et al., 2020), ERNIE (Zhang et al., 2019b) and ProphetNet (Qi et al., 2021), or an objective specifically tailored for summarization like in PEGASUS (Zhang et al., 2020). Most of these sequence-to-sequence models are history-based, where an output sequence is repre- sented as a sequence of decisions and the probabil- *Equal contribution. Decoding methods # Summary candidates R-1 R-2 R-L BS BaS Beam search (top beam) 1 44.23 21.48 41.21 87.39 -2.78 Beam search 15 51.06 27.74 48.05 88.50 -2.48 Diverse beam search 15 54.30 30.02 51.33 88.97 -2.40 Top-k sampling 15 52.31 27.41 49.17 88.64 -2.56 Top-p sampling 15 53.52 28.88 50.46 88.87 -2.46 Adding all four methods above 60 57.70 33.77 54.72 89.58 -2.25 Table 1: Oracle scores (maximum scores over all gener- ated candidates) for four popular decoding methods and five summarization evaluation measures for a base PE- GASUS model on CNN/DM. R-1/2/L denotes ROUGE- 1/2/L, BS and BaS denote BERTScore and BARTScore, respectively. ity of the sequence is computed as a product of de- cision probabilities. This is also known as the auto- regressive factorization. To transform the sequence of probabilities into summaries, beam search is commonly used. While auto-regressive decoding with beam search is simple and has many advan- tages, it can be difficult to encode global constraints such as grammaticality, coherence and factual …

Content truncated to 500 words