Improving Robust Domain Independent Summarization

Improving Robust Domain Independent Summarization Jim Cowie, Eugene Ludovik, Hugo Molina-Salgado Dept. 3CRL, Box 30001, NMSU, Las Cruces, NM 88003, USA (jcowie, eugene, hsalgado) @crl.nmsu.edu Abstract We discuss those techniques which, in the opinion of the authors, are needed to support robust automatic summarization. Many of these methods are already incorporated in a multi-lingual summarization system, MINDS, developed at CRL. The approach is sentence selection, but includes techniques to improve coherence and also to perform sentence reduc- tion. Our methods are in distinct contrast to those approaches to summarization by deep analysis of a document followed by text gener- ation. KEYWORDS Summarization, Multi-lingual Language Engi- neering, Robust Methods 1 Introduction Summarization is the problem of presenting the most important information contained in one or more documents. The research described here focuses on multi-lingual summarization (MLS). Summaries of documents are produced in Spanish, Japanese, English and Russian using the same basic summarization engine. The core summarization problem is taking a single text and producing a shorter text in the same language that contains all the main points in the input text. We are using a robust, graded approach to building the core engine by incorporating statis- tical, syntactic and document structure analyses among other techniques. We have developed a sys- tem design which allows the parameterization both of the summarization process and of necessary information about the languages being processed. Document structure analysis (Salton & Singal 94, Salton et al. 95) is important for extracting the topic of a text. In a statistical analysis for example (Paice 90, Paice & Jones 93), titles and sub-titles would be given a more important weight than the body of the text. Similarly, the introduction and conclusion for the text itself and for each section are more important than other paragraphs, and the first and last sentences in each paragraph are more important than others. The applicability of these depends on the style adopted in a particular domain, and on the language: the stylistic structure and the presentation of arguments vary signifi- cantly across genres and languages. Structure anal- ysis must be tailored to a particular type of text in a particular language. In the MINDS system docu- ment structure analysis involves the following sub- tasks: • Language Identification • Document Structure Parsing • Multilingual Sentence Segmentation • Text Structure Heuristics In order to allow a multitude of techniques to contribute to sentence selection, the core engine adopts a flexible method of scoring the sentences in a document by each of the techniques and then ranking them by combining the different scores. Text-structure based heuristics provide the main method for ranking and selecting sentences in a document. These are supplemented by word fre- quency analysis methods. The core engine is designed in such a way that as additional resources, such as lexical and other knowledge bases or text processing and MT engines, become available from other ongoing research efforts they can be incorporated into the overall multi-engine MINDS system. The most promising components are part …

Content truncated to 500 words