SUM-QE: a BERT-based Summary Quality Estimation Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6005–6011, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6005 SUM-QE: a BERT-based Summary Quality Estimation Model Stratos Xenouleas1, Prodromos Malakasiotis1, Marianna Apidianaki2 and Ion Androutsopoulos1 1 Department of Informatics, Athens University of Economics and Business, Greece 2 CNRS, LLF, France and University of Helsinki, Finland stratosxen@gmail.com, rulller@aueb.gr marianna.apidianaki@helsinki.fi, ion@aueb.gr Abstract We propose SUM-QE, a novel Quality Es- timation model for summarization based on BERT. The model addresses linguistic qual- ity aspects that are only indirectly captured by content-based approaches to summary evalu- ation, without involving comparison with hu- man references. SUM-QE achieves very high correlations with human ratings, outperform- ing simpler models addressing these linguis- tic aspects. Predictions of the SUM-QE model can be used for system development, and to in- form users of the quality of automatically pro- duced summaries and other types of generated text. 1 Introduction Quality Estimation (QE) is a term used in machine translation (MT) to refer to methods that measure the quality of automatically translated text without relying on human references (Bojar et al., 2016, 2017). In this study, we address QE for summa- rization. Our proposed model, SUM-QE, success- fully predicts linguistic qualities of summaries that traditional evaluation metrics fail to capture (Lin, 2004; Lin and Hovy, 2003; Papineni et al., 2002; Nenkova and Passonneau, 2004). SUM-QE pre- dictions can be used for system development, to inform users of the quality of automatically pro- duced summaries and other types of generated text, and to select the best among summaries out- put by multiple systems. SUM-QE relies on the BERT language repre- sentation model (Devlin et al., 2019). We use a pre-trained BERT model adding just a task- specific layer, and fine-tune the entire model on the task of predicting linguistic quality scores manually assigned to summaries. The five cri- teria addressed are given in Figure 1. We pro- vide a thorough evaluation on three publicly avail- able summarization datasets from NIST shared Q1 – Grammaticality: The summary should have no datelines, system-internal formatting, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. Q2 – Non redundancy: There should be no unnecessary repetition in the summary. Q3 – Referential Clarity: It should be easy to identify who or what the pronouns and noun phrases in the sum- mary are referring to. Q4 – Focus: The summary should have a focus; sentences should only contain information that is related to the rest of the summary. Q5 – Structure & Coherence: The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of informa- tion about a topic. Figure 1: SUM-QE rates summaries with respect to five linguistic qualities (Dang, 2006a). The datasets we use for tuning and evaluation contain human assigned scores …

Content truncated to 500 words