Utility Evaluation of Cross-document Information Extraction

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 285–288, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Utility Evaluation of Cross-document Information Extraction � Heng Ji a, Zheng Chen a, Jonathan Feldman a, Antonio Gonzalez a, Ralph Grishman b, Vivek Upadhyay a aComputer Science Department, Queens College and the Graduate Center, City University of New York New York, NY 11367, USA bComputer Science Department, New York University, New York, NY 10003, USA hengji@cs.qc.cuny.edu, zchen1@gc.cuny.edu, agonzalez117@qc.cuny.edu, grishman@cs.nyu.edu, vivekqc@gmail.com Abstract We describe a utility evaluation to determine whether cross-document information extrac- tion (IE) techniques measurably improve user performance in news summary writing. Two groups of subjects were asked to perform the same time-restricted summary writing tasks, reading news under different conditions: with no IE results at all, with traditional single- document IE results, and with cross-document IE results. Our results show that, in compari- son to using source documents only, the qual- ity of summary reports assembled using IE results, especially from cross-document IE, was significantly better and user satisfaction was higher. We also compare the impact of different user groups on the results. 1 Introduction Information Extraction (IE) is a task of identifying ‘‘facts’’ (entities, relations and events) within un- structured documents, and converting them into structured representations (e.g., databases). IE techniques have been effectively applied to differ- ent domains (e.g. daily news, Wikipedia, biomedi- cal reports, financial analysis and legal documentations) and different languages. Recently we described a new cross-document IE task (Ji et al., 2009) to extract events across-documents and track them on a time line. Compared to traditional single-document IE, this new task can extract more salient, accurate and concise event information. However, a significant question remains: will the events extracted by IE, especially this new cross-document IE task, actually help end-users to make better use of the large volumes of news? In order to investigate whether we have reached this goal, we performed an extrinsic utility (i.e., use- fulness) and usability evaluation on IE results. Two groups of subjects were asked to perform the same time-restricted summary writing tasks, read- ing news under different conditions: with no IE results at all, with traditional single-document IE results, and with cross-document IE results. Our results show that, in comparison to using source documents only, the quality of summary reports assembled using IE techniques, especially from cross-document IE, was significantly better. Also, as extraction quality increases from no IE at all to single-document IE and then to cross-document IE, user satisfaction increases. We also compare the impact of different user groups on the results. To the best of our knowledge, this is the first system- atic evaluation of cross-document IE from a us- ability perspective. 2 Overview of IE Systems We applied the English single-document IE system (Ji and Grishman, 2008) and cross-document IE system presented in (Ji et al., 2009). Both systems were developed for the ACE program1. The single-document IE system can extract events from individual documents. The …

Content truncated to 500 words