An Evaluation Summary Method Based on a Combination of Content and Linguistic Metrics

Proceedings of Recent Advances in Natural Language Processing, pages 245–251, Hissar, Bulgaria, 7-13 September 2013. An Evaluation Summary Method Based on a Combination of Content and Linguistic Metrics Samira Ellouze University of Sfax ANLP Research Group, MIRACL Laboratory, Sfax, Tunisia ellouze.samira@gmail .com Maher Jaoua University of Sfax ANLP Research Group, MIRACL Laboratory, Sfax, Tunisia Maher.Jaoua@fsegs.rn u.tn Lamia Hadrich Belguith University of Sfax ANLP Research Group, MIRACL Laboratory, Sfax, Tunisia l.belguith@fsegs.rn u.tn Abstract This paper presents a new automated method for evaluating the content of a text summary. The proposed method is based on a combination of features encompassing scores of content and others of linguistic quality. This method relies on a learning technique called linear regression. The objective of this combination is to predict the PYRAMID score from the features used. In order to evaluate the presented method, we are interested in two levels of granularity evaluation: the first is named Micro- evaluation and proposes an evaluation of each summary while the second is called Macro-evaluation and it is applied at the level of each system. The resulting metric shows an improvement upon standard metrics by increasing the correlation with the PYRAMID metric. 1 Introduction The evaluation of a summary is an important and necessary task. It quantifies the informativeness and linguistic quality of a summary and it can be of two types: extrinsic or intrinsic (Jing et al., 1998). Extrinsic evaluation measures the impact of using a summary in the place of the source document(s) on tasks such as document classification and indexing while intrinsic evaluation assesses the overall quality of the summary either manually or automatically. It should be noted that the manual evaluation is a difficult and expensive task because it requires a lot of time and expertise in the field of the source text topic. For this reason, several automatic evaluation metrics have been developed such as ROUGE (Lin, 2004), BE (Hovy et al., 2006), BEwTE (Tratz and Hovy, 2008), AutoSummENG (Giannakopoulos et al., 2008), etc. The advent of automatic evaluation metrics generates in its turn a new step: meta-evaluation i.e. the evaluation of evaluation metrics. We perform this meta-evaluation by making a comparison between these metrics and manual metrics. To achieve this comparison, the TAC1 conference proposed various metrics of correlations (i.e. Pearson, Spearman, Kandall). Most of the evaluation metrics assessed by the TAC conference are based on the evaluation of the relevance of a summary content. However, a summary with relevant content may be unreadable. To encourage researchers to evaluate the readability of a summary, the TAC 2011 session added a new goal to the task of automatic evaluation of summaries consisting in evaluating the readability of summaries. In this context, we suggest in this paper an evaluation method based on the combination of several evaluation metrics (i.e. content metrics and linguistic quality metrics). This paper is organized as follows: in section 2, we give a brief historical overview on the evolution of the evaluation of intrinsic methods used in the field of automatic …

Content truncated to 500 words