Corpus and Evaluation Measures for Multiple Document Summarization with Multiple Sources

Corpus and Evaluation Measures for Multiple Document Summarization with Multiple Sources Tsutomu HIRAO NTT Communication Science Laboratories hirao@cslab.kecl.ntt.co.jp Takahiro FUKUSIMA Otemon Gakuin University fukusima@res.otemon.ac.jp Manabu OKUMURA Tokyo Institute of Technology oku@pi.titech.ac.jp Chikashi NOBATA Communication Research Laboratories nova@crl.go.jp Hidetsugu NANBA Hiroshima City University nanba@its.hiroshima-cu.ac.jp Abstract In this paper, we introduce a large-scale test collec- tion for multiple document summarization, the Text Summarization Challenge 3 (TSC3) corpus. We detail the corpus construction and evaluation mea- sures. The significant feature of the corpus is that it annotates not only the important sentences in a doc- ument set, but also those among them that have the same content. Moreover, we define new evaluation metrics taking redundancy into account and discuss the effectiveness of redundancy minimization. 1 Introduction It has been said that we have too much informa- tion on our hands, forcing us to read through a great number of documents and extract relevant informa- tion from them. With a view to coping with this situ- ation, research on automatic text summarization has attracted a lot of attention recently and there have been many studies in this field. There is a particular need to establish methods for the automatic sum- marization of multiple documents rather than single documents. There have been several evaluation workshops on text summarization. In 1998, TIPSTER SUM- MAC (Mani et al., 2002) took place and the Doc- ument Understanding Conference (DUC)1 has been held annually since 2001. DUC has included multi- ple document summarization among its tasks since the first conference. The Text Summarization Chal- lenge (TSC)2 has been held once in one and a half years as part of the NTCIR (NII-NACSIS Test Col- lection for IR Systems) project since 2001. Multiple document summarization was included for the first time as one of the tasks at TSC2 (in 2002) (Okumura et al., 2003). Multiple document summarization is now a central issue for text summarization research. 1http://duc.nist.gov 2http://www.lr.pi.titech.ac.jp/tsc In this paper, we detail the corpus construction and evaluation measures used at the Text Summa- rization Challenge 3 (TSC3 hereafter), where multi- ple document summarization is the main issue. We also report the results of a preliminary experiment on simple multiple document summarization sys- tems. 2 TSC3 Corpus 2.1 Guidelines for Corpus Construction Multiple document summarization from multiple sources, i.e., several newspapers concerned with the same topic but with different publishers, is more dif- ficult than single document summarization since it must deal with more text (in terms of numbers of characters and sentences). Moreover, it is peculiar to multiple document summarization that the sum- marization system must decide how much redun- dant information should be deleted3. In a single document, there will be few sentences with the same content. In contrast, in multiple doc- uments with multiple sources, there will be many sentences that convey the same content with differ- ent words and phrases, or even identical sentences. Thus, a text summarization system needs to recog- nize such redundant sentences and reduce the redun- dancy in the output summary. However, …

Content truncated to 500 words