The effects of analysing cohesion on document summarisation

The effects of analysing cohesion on document summarisation Branimir K. Boguraev and Mary S. Neff IBM T.J. Watson Research Center, P.O. Box 704, Yorktown Heights, NY 10598, USA bkb,neff@watson.ibm.com Abstract We argue that in general, the analysis of lexical co- hesion factors in a document can drive a summarizer, as well as enable other content characterization tasks. More narrowly, this paper focuses on how one particular cohesion factor—simple lexical repetition—can enhance an existing sentence extraction summarizer, by enabling strategies for overcoming some particularly jarring end- user effects in the summaries, typically due to coher- ence degradation, readability deterioration, and topical under-representation. Lexical repetition is instrumental to, among other things, the topical make-up of a text, and in our framework a lexical repetition-based model of dis- course segmentation, capable of detecting topic shifts, is integrated with a linguistically-aware summarizer utiliz- ing notions of salience and dynamically-adjustable sum- mary size. We show that even by leveraging lexical rep- etition alone, summaries are of comparable, and under certain conditions better, quality than the ones delivered by a state-of-the-art summarizer. This is encouraging for a broad research platform focusing on the recognition and use of cohesive devices in text for a range of content characterisation and document management tasks. 1 Introduction This paper addresses a particular class of problems inher- ent to summaries derived by sentence extraction, namely the related issues of coherence degradation, readability de- terioration, and topical under-representation. Fundamen- tally, these problems arise from unconstrained deletion of arbitrary amount of source material between two sen- tences which end up adjacent in the summary; this has unpredictable effects on the amount of potentially essen- tial information which may be lost in that deletion. Ex- amples like ‘dangling’ anaphors (with lost antecedents) have been cited often enough, and strategies like includ- ing the immediately preceding sentence in the summary have some effect. While intuitively plausible, these are still simple strategies, prone to misfiring; moreover, other effects like the reversal of a core premise in an argument, or the introduction, and subsequent elaboration, of a new topic, are not easily handled by similar heuristics. We seek to leverage a mechanism for assessing the de- gree of cohesion between individual sentences in the source document, as well as having a notion of how these map onto the underlying themes in the document. Informally, cohesion—and lexical cohesion in particular—is manifest in the ways in which the words, or word patterns, of a sentence connect that sentence to certain of its predeces- sors and successors. The intuition is that identifying, and preserving, some of these connections in the summary would improve its coherence. 1.1 Lexical cohesion and summarization Documents are coherent because of the continuity of their discourse. A number of rhetorical devices help achieve cohesion between related document fragments. Analysing such devices—or at the very least being sen- sitive to their manifestation and interplay—can bring a moderately refined degree of discourse awareness into the summarization process. In the absence of deep text understanding, this boils down to …

Content truncated to 500 words