Detecting Shifts in News Stories for Paragraph Extraction

Detecting Shifts in News Stories for Paragraph Extraction Fumiyo Fukumoto Yoshimi Suzuki Department of Computer Science and Media Engineering, Yamanashi University 4-3-11, Takeda, Kofu, 400-8511, Japan {fukumoto@skye.esb, ysuzuki@alps1.esi}.yamanashi.ac.jp Abstract For multi-document summarization where docu- ments are collected over an extended period of time, the subject in a document changes over time. This paper focuses on subject shift and presents a method for extracting key paragraphs from documents that discuss the same event. Our extraction method uses the results of event tracking which starts from a few sample documents and finds all subsequent docu- ments that discuss the same event. The method was tested on the TDT1 corpus, and the result shows the effectiveness of the method. 1 Introduction Multi-document summarization of news stories dif- fers from single document in that it is important to identify differences and similarities across doc- uments. This can be interpreted as the question of how to identify an event and a subject in doc- uments. According to the TDT project, an event is something that occurs at a specific place and time as- sociated with some specific actions, and it becomes the background among documents. A subject, on the other hand, refers to a theme of the document itself. Another important factor, which is typical in a stream of news, is recognizing and handling sub- ject shift. The extracted paragraphs based on an event and a subject may include the main points of each document and the background among docu- ments. However, when they are strung together, the resulting summary still contains much overlapping information. This paper focuses on subject shift and presents a method for extracting key paragraphs from docu- ments that discuss the same event. We use the re- sults of our tracking technique which automatically detects subject shift, and produces the optimal win- dow size in the training data so as to include only the data which are sufficiently related to the current sub- ject. The idea behind this is that, of two documents from the target event which are close in chronological order, the latter discusses (i) the same subject as an earlier one, or (ii) a new subject related to the tar- get event. This is particularly well illustrated by the Kobe Japan quake event in the TDT1 data. The first document says that a severe earthquake shook the city of Kobe. It continues until the 5th document. The 6th through 17th documents report damage, lo- cation and nature of quake. The 18th document, on the other hand, states that the Osaka area suffered much less damage than Kobe. The subject of the document is different from the earlier ones, while all of these are related to the Kobe Japan quake event. We use the leave-one-out estimator of Support Vec- tor Machines(SVMs)(Vapnik, 1995) to make a clear distinction between (i) and (ii) and thus estimate the optimal window size in the training data. For the re- sults of tracking where documents are divided into several sets, each of which covers …

Content truncated to 500 words