Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings

Proceedings of the 27th International Conference on Computational Linguistics, pages 905–914 Santa Fe, New Mexico, USA, August 20-26, 2018. 905 Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, Fang Chen University of New South Wales, Sydney, Australia Data61 CSIRO, Sydney, Australia {elahehs,mohammade,wong,fang}@cse.unsw.edu.au Abstract We present a new summary evaluation approach that does not require human model summaries. Our approach exploits the compositional capabilities of corpus-based and lexical resource-based word embeddings to develop the features reflecting coverage, diversity, informativeness, and coherence of summaries. The features are then used to train a learning model for predicting the summary content quality in the absence of gold models. We evaluate the proposed metric in replicating the human assigned scores for summarization systems and summaries on data from query-focused and update summarization tasks in TAC 2008 and 2009. The results show that our feature combination provides reliable estimates of summary content quality when model summaries are not available. 1 Introduction Quantifying the quality of summaries is an important and necessary task in the field of automatic text summarization. Current summary evaluation methods like manual and automated pyramid (Passon- neau et al., 2005; Passonneau et al., 2013) and well-established ROUGE scores (Lin, 2004) heavily rely on multiple human-generated model summaries to assess the quality of system-generated summaries. This evaluation paradigm falls short on non-standard test sets where model summaries are not available. According to the quantitative analysis by Louis and Nenkova (2009a); Singh and Jin (2016), evaluat- ing summaries by their comparison with the input obtains good correlations with manual evaluations. Therefore, identifying a suitable input-summary similarity metric will provide a means for model-free evaluation of summaries. We hypothesize that comparing semantic representations of the input and summary content will lead to more accurate input-summary evaluation. Hence, we explore the effectiveness of compositionality of word embeddings in developing a model-free automatic metric to evaluate summary content quality. In particular, our approach incorporates the word embedding models trained on the Google News corpus and the WordNet lexical resource to compare centroid vectors of the input and summary. To demonstrate the effectiveness of our approach, we have conducted a set of experiments on data from query-focused and update summarization tasks in TAC1 2008 and 2009. The reliability of our metric is also studied conducting an error analysis. The experiment results show that quantifying the indicators of content quality by taking advantage of compositional properties of the word and sense embeddings produces summary scores which accurately replicate human assessments. It is noteworthy that our approach com- plements but is not intended to replace existing model-based evaluation approaches, since their reliability and strength are important for high confidence evaluations. 2 Related Work Proposals for developing automatic summary evaluation methods (Ellouze et al., 2013; Ng and Abrecht, 2015; ShafieiBavani et al., 2017) have been put forward in the past. However, these methods are not This work is licensed under a Creative Commons Attribution 4.0 International License. License details: …

Content truncated to 500 words