Supervised Learning of Automatic Pyramid for Optimization-Based Multi-Document Summarization

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1084–1094 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1100 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1084–1094 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1100 Supervised Learning of Automatic Pyramid for Optimization-Based Multi-Document Summarization Maxime Peyrard and Judith Eckle-Kohler Research Training Group AIPHES and UKP Lab Computer Science Department, Technische Universit¨at Darmstadt www.aiphes.tu-darmstadt.de, www.ukp.tu-darmstadt.de Abstract We present a new supervised framework that learns to estimate automatic Pyramid scores and uses them for optimization- based extractive multi-document summa- rization. For learning automatic Pyramid scores, we developed a method for au- tomatic training data generation which is based on a genetic algorithm using auto- matic Pyramid as the fitness function. Our experimental evaluation shows that our new framework significantly outperforms strong baselines regarding automatic Pyra- mid, and that there is much room for im- provement in comparison with the upper- bound for automatic Pyramid. 1 Introduction We consider extractive text summarization, the task of condensing a textual source, e.g., a set of source documents in multi-document summariza- tion (MDS), into a short summary text. The qual- ity of an automatic system summary is tradition- ally evaluated by comparing it against one or more reference summaries written by humans. This comparison is performed by means of an evalua- tion metric measuring indicators of summary qual- ity and combining them into an aggregated score. Many state-of-the-art summarization systems cast extractive summarization as an optimization problem and maximize an objective function in or- der to create good, i.e., high-scoring summaries. To this end, optimization-based systems com- monly use an objective function which encodes exactly those quality indicators which are mea- sured by the particular evaluation metric being used. Some systems even employ an approxima- tion of the evaluation metric as objective function. Consider as an example the ROUGE metric which has become a de-facto standard for sum- mary evaluation (Lin, 2004). ROUGE computes the n-gram overlap between a system summary and a pool of reference summaries. There are sev- eral previous approaches which have used an ap- proximation of ROUGE as the optimization objec- tive (e.g., Sipos et al. (2012); Peyrard and Eckle- Kohler (2016a)). However, ROUGE has been widely criticized for being too simplistic and not suitable for captur- ing important quality aspects we are interested in. In particular, ROUGE does not capture sentences which are semantically equivalent but expressed with different words (Nenkova et al., 2007). Ideally, we would like to evaluate our sum- maries based on human judgments. A well-known example of such a human evaluation method is the so-called Pyramid method (Nenkova et al., 2007): it evaluates the particular quality aspect of content selection and is based on a manual comparison of Summary Content Units (SCUs) in reference sum- maries against SCUs in system summaries. While the resulting Pyramid score is much more mean- ingful and informative than ROUGE, it …

Content truncated to 500 words