Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3612–3621, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3612 Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning Hanlu Wu1∗, Tengfei Ma2∗, Lingfei Wu2, Tariro Manyumwa1, Shouling Ji1 1Zhejiang University, 2 IBM Research wuhanlu@zju.edu.cn, tengfei.ma1@ibm.com,wuli@us.ibm.com, tmanyumwa@yahoo.com, sji@zju.edu.cn Abstract Evaluation of a document summarization sys- tem has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we pro- pose to evaluate the summary qualities with- out reference summaries by unsupervised con- trastive learning. Specifically, we design a new metric which covers both linguistic qual- ities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative sam- ples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Further- more, we show that our method is general and transferable across datasets. 1 Introduction Recently, there has been great success in automatic text summarization and generation (Huang et al., 2020; LeClair et al., 2020; Chen et al., 2020). To better compare and improve the performance of models, evaluation for such systems has been a problem of interest. The selection of evaluation metrics will greatly affect the assessed quality of a generated summary and thus affect the evaluation of summarization models. The most ideal metric is definitely human judge- ment, which is often treated as the gold standard. But human evaluation is time-consuming and labor- intensive, an automatic evaluation metric that can- not only save human resources but also simulate the ability of human judgement is of crucial impor- tance. ∗Equal contribution Most of the existing automatic evaluation meth- ods assess a summary by comparing it with ref- erence texts written by humans. Some of them are model-free and simply use hand-crafted match- ing functions to calculate the similarity between the candidate summary and the reference (Papineni et al., 2002; Lin and Och, 2004; Banerjee and Lavie, 2005). These methods consider both the reference and the candidate as a sequence of tokens or n- gram blocks. For instance, as the de facto standard evaluation metric, ROUGE (Lin and Och, 2004) calculates the n-gram overlap between the machine- generated summaries and reference summaries. Al- though these methods have the advantage of inter- pretability and efficiency, they are found to corre- late poorly with human evaluation (Novikova et al., 2017). To reduce the requirement of exact word match- ing, some recent work tried to match the refer- ence and the candidate summary in the embed- ding space of words or sentences (Zhang et al., 2020; Clark et al., 2019; Zhao et al., 2019). For instance, BERTScore (Zhang et al., 2020) uses con- textual word embeddings generated by BERT and performs a …

Content truncated to 500 words