Summarization and Evaluation; Where are we today?!

Summarization and Evaluation; Where are we today?!∗ Mehrnoush Shamsfarda, Amir Saffariana, Samaneh Ghodratnamab a NLP Laboratory, Electrical and Computer Engineering Department, Shahid Beheshti University, Velenjak, Tehran, Iran m-shams@sbu.ac.ir a_saffarian@std.sbu.ac.ir b Department of Computer Science and Engineering, Shiraz University, Molasadra, Shiraz, Iran ghodratnama@cse.shirazu.ac.ir Abstract. The rapid growth of the online information services causes the problem of information explosion. Automatic text summarization techniques are essential for dealing with this problem. There are different approaches to text summarization and different systems have used one or a combination of them. Considering the wide variety of summarization techniques there should be an evaluation mechanism to assess the process of summarization. The evaluation of automatic summarization is important and challenging, since in general it is difficult to agree on an ideal summary of a text. Currently evaluating summaries is a laborious task that could not be done simply by human so automatic evaluation techniques are appearing to help this matter. In this paper, we will take a look at summarization approaches and examine summarizers’ general architecture. The importance of evaluation methods is discussed and the need to find better automatic systems to evaluate summaries is studied. Keywords: Summarization, Evaluation, Recall, Precision, Pyramid, ROUGE 1. Introduction Today, having access to information summaries is one of the important needs for humans which may affect their lives. These summaries generally are produced with the help of other humans. The question arises here is that in this era where we are encountering a huge volume of increasing data and information, with limited resources to decide on, may it be really possible to use humans as information summarizers? We believe not. So, researchers have created methods to automatically do summarization task and extract the most important concepts of the information without any need to involve humans. Now a second question emerges. How could we know about the correctness and the completeness of the results generated in an automatic manner? The answer is hidden in evaluation techniques on which we are going to focus in this paper. Therefore besides the growth of summarization methods, evaluation techniques must improve and mature so as to provide acceptable scores for summaries according to some sort of evaluation metrics. Evaluation approaches mostly evaluate created summaries based on (1) ideal (gold) summary, (2) use in an application and (3) original document. Techniques from the first category compare system generated summaries with a summary known as the best possible summary! This ideal summary currently is created by humans and it could be influenced by subjective effects from judges. Application driven techniques evaluate content of summary by analyzing the level of information that could be obtained from it for a specific task. In this approach the performance of the application is analyzed when using the original document and ∗ Copyright 2007 by Mehrnoush Shamsfard, Amir Saffarian, Samaneh Ghodratnama 422 its summary separately. On the other hand evaluation by original document is really ambiguous because specifying evaluation parameters and metrics is hard but it is more natural. In this paper …

Content truncated to 500 words