Who wrote What Where: Analyzing the content of human and automatic summaries

Proceedings of the Workshop on Automatic Summarization for Different Genres, Media, and Languages, pages 25–32, Portland, Oregon, June 23, 2011. c⃝2011 Association for Computational Linguistics Who wrote What Where: Analyzing the content of human and automatic summaries Karolina Owczarzak and Hoa Trang Dang Information Access Division National Institute of Standards and Technology Gaithersburg, MD 20899 karolina.owczarzak@nist.gov hoa.dang@nist.gov Abstract Abstractive summarization has been a long- standing and long-term goal in automatic sum- marization, because systems that can generate abstracts demonstrate a deeper understanding of language and the meaning of documents than systems that merely extract sentences from those documents. Genest (2009) showed that summaries from the top automatic sum- marizers are judged as comparable to manual extractive summaries, and both are judged to be far less responsive than manual abstracts, As the state of the art approaches the limits of extractive summarization, it becomes even more pressing to advance abstractive summa- rization. However, abstractive summarization has been sidetracked by questions of what qualifies as important information, and how do we find it? The Guided Summarization task introduced at the Text Analysis Conference 2010 attempts to neutralize both of these prob- lems by introducing topic categories and lists of aspects that a responsive summary should address. This design results in more similar human models, giving the automatic summa- rizers a more focused target to pursue, and also provides detailed diagnostics of summary con- tent, which can can help build better meaning- oriented summarization systems. 1 Introduction What qualifies as important information and how do we find it? These questions have been leading re- search in automatic summarization since its begin- nings, and we are still nowhere near a definitive answer. Worse, experiments with humans subjects suggest a definitive answer might not even exist. With all their near-perfect language understanding and world knowledge, two human summarizers will still produce two different summaries of the same text, simply because they will disagree on what’s important. Fortunately, usually some of this infor- mation will overlap. This is represented by the idea behind the Pyramid evaluation framework (Nenkova and Passonneau, 2004; Passonneau et al., 2005), where different levels of the pyramid represent the proportion of concepts (“Summary Content Units”, or SCUs) mentioned by 1 to n summarizers in sum- maries of the same text. Usually, there are very few SCUs that are mentioned by all summarizers, a few more that are mentioned by some of them, and the greatest proportion are the SCUs that are mentioned by individual summarizers only. This variance in what should be a “gold standard” makes research in automatic summarization meth- ods particularly difficult. How can we reach a goal so vague and under-defined? Using term frequency to determine important concepts in a text has proven to be very successful, largely because of its simplic- ity and universal applicability, but statistical meth- ods can only provide the most basic level of perfor- mance. On the other hand, there is no real motiva- tion to use any deeper meaning-oriented text anal- ysis if …

Content truncated to 500 words