Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue

Findings of the Association for Computational Linguistics: ACL 2022, pages 4119 - 4131 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue Alexios Gidiotis Aristotle University of Thessaloniki gidiotis@csd.auth.gr Grigorios Tsoumakas Aristotle University of Thessaloniki greg@csd.auth.gr Abstract We explore the notion of uncertainty in the context of modern abstractive summariza- tion models, using the tools of Bayesian Deep Learning. Our approach approximates Bayesian inference by first extending state- of-the-art summarization models with Monte Carlo dropout and then using them to perform multiple stochastic forward passes. Based on Bayesian inference we are able to effectively quantify uncertainty at prediction time. Hav- ing a reliable uncertainty measure, we can im- prove the experience of the end user by fil- tering out generated summaries of high un- certainty. Furthermore, uncertainty estimation could be used as a criterion for selecting sam- ples for annotation, and can be paired nicely with active learning and human-in-the-loop ap- proaches. Finally, Bayesian inference enables us to find a Bayesian summary which performs better than a deterministic one and is more robust to uncertainty. In practice, we show that our Variational Bayesian equivalents of BART and PEGASUS can outperform their deterministic counterparts on multiple bench- mark datasets. 1 Introduction State-of-the-art text summarization methods have achieved remarkable performance in various bench- marks (Song et al., 2019; Dong et al., 2019; Lewis et al., 2019; Zhang et al., 2020). The majority of these methods use very large Transformer models pre-trained on language generation tasks. Although such methods can generate high qual- ity summaries for texts similar to their training set, they suffer from a couple of issues when the inputs lie far from the training data distribution. They are prone to generating particularly bad outputs (Xu et al., 2020; Kry´sci´nski et al., 2020) and are usually fairly confident about them (Gal and Ghahramani, 2016; Xiao et al., 2020). These shortcomings are bound to cause problems once a summarization model is deployed to solve a practical problem. Since the output of automatic summarization models is usually expected to be consumed by hu- mans, it is very important to know when such an output is of good enough quality to be served to users. In most cases, it is very much preferable to not serve an output at all, instead of serving a bad output. This will in turn increase users’ trust to automated summarization systems. Model uncertainty is one way of detecting when a model’s output is likely to be poor on the grounds of predicting far away from it’s training distribu- tion. Recent summarization methods have focused heavily on improving the overall performance, but model uncertainty has been explored very little (Xu et al., 2020). In addition to improving user experience, the development of uncertainty measures for summa- rization can pave the way for active learning ap- proaches (Gal et al., 2017; Houlsby et al., 2011; Liu et al., 2020; Lyu et al., 2020). The value of active learning stems from …

Content truncated to 500 words