Evaluating Information Content by Factoid Analysis: Human annotation and stability

Evaluating information content by factoid analysis: human annotation and stability Simone Teufel Computer Laboratory Cambridge University, UK Hans van Halteren Language and Speech University of Nijmegen, The Netherlands Abstract We present a new approach to intrinsic sum- mary evaluation, based on initial experiments in van Halteren and Teufel (2003), which com- bines two novel aspects: comparison of infor- mation content (rather than string similarity) in gold standard and system summary, mea- sured in shared atomic information units which we call factoids, and comparison to more than one gold standard summary (in our data: 20 and 50 summaries respectively). In this paper, we show that factoid annotation is highly re- producible, introduce a weighted factoid score, estimate how many summaries are required for stable system rankings, and show that the fac- toid scores cannot be sufficiently approximated by unigrams and the DUC information overlap measure. 1 Introduction Many researchers in summarisation believe that the best way to evaluate a summary is extrin- sic evaluation (Sp¨arck Jones, 1999): to measure the quality of the summary on the basis of de- gree of success in executing a specific task with that summary. The summary evaluation per- formed in SUMMAC (Mani et al., 1999) fol- lowed that strategy. However, extrinsic eval- uations are time-consuming to set up and can thus not be used for the day-to-day evaluation needed during system development. So in prac- tice, a method for intrinsic evaluation is needed, where the properties of the summary itself are examined, independently of its application. Intrinsic evaluation of summary quality is un- deniably hard, as there are two subtasks of sum- marisation which need to be evaluated, infor- mation selection and text production — in fact these two subtasks are often separated in evalu- ation (Mani, 2001). If we restrict our attention to information selection, systems are tested by way of comparison against a “gold standard”, a manually produced result which is supposed to be the “correct”, “true” or “best” result. In summarisation there appears to be no “one truth”, but rather various “good” results. Hu- man subjectivity in what counts as the most im- portant information is high. This is evidenced by low agreement on sentence selection tasks (Rath et al., 1961; Jing et al., 1998), and low word overlap measures in the task of creating summaries by reformulation in the summaris- ers’ own words (e.g. word overlap of the 542 single document summary pairs in DUC-02 av- eraged only 47%). But even though the non-existence of any one gold standard is generally acknowledged in the summarisation community, actual practice nev- ertheless ignores this, mostly due to the expense of compiling summary gold standards and the lack of composite measures for comparison to more than one gold standard. Other fields such as information retrieval (IR) also have to deal with human variability con- cerning the question of what “relevant to a query” means. This problem is circumvented by extensive sampling: many different queries are collected to level out the differences in query formulation …

Content truncated to 500 words