Evaluating the Portability of Revision Rules for Incremental Summary Generation

Evaluating the Portability of Revision Rules for Incremental Summary Generation Jacques Robin http://www.di.ufpe.br/~jr jr@di.ufpe.br Departamento de Inform£tica, Universidade Federal de Pernambuco Caixa Postal 7851, Cidade Universit£ria Recife, PE 50732-970 Brazil Abstract This paper presents a quantitative evalu- ation of the portability to the stock mar- ket domain of the revision rule hierarchy used by the system STREAK to incremen- tally generate newswire sports summaries. The evaluation consists of searching a test corpus of stock market reports for sentence pairs whose (semantic and syntactic) struc- tures respectively match the triggering con- dition and application result of each revi- sion rule. The results show that at least 59% of all rule classes are fully portable, with at least another 7% partially portable. 1 Introduction The project STREAK 1 focuses on the specific issues involved in generating short, newswire style, natural language texts that summarize vast amount of in- put tabular data in their historical context. A series of previous publications presented complementary aspects of this project: motivating corpus analysis in (Robin and McKeown, 1993), new revision-based text generation model in (Robin, 1993), system im- plementation and rule base in (Robin, 1994a) and empirical evaluation of the robustness and scalabil- ity of this new model as compared to the traditional single pass pipeline model in (Robin and McKeown, 1995). The present paper completes this series by describing a second, empirical, corpus-based evalu- ation, this time quantifying the portability to an- other domain (the stock market) of the revision rule hierarchy acquired in the sports domain and imple- mented in STREAK. The goal of this paper is twofold: (1) assessing the generality of this particular rule hi- erarchy and (2) providing a general, semi-automatic 1 Surface Text Reviser Expressing Additional Knowledge. methodology for evaluating the portability of seman- tic and syntactic knowledge structures used for nat- ural language generation. The results reveal that at least 59% of the revision rule hierarchy abstracted from the sports domain could also be used to incre- mentally generate the complex sentences observed in a corpus of stock market reports. I start by providing the context of the evalua- tion with a brief overview of STREAK's revision-based generation model, followed by some details about the empirical acquisition of its revision rules from cor- pus data. I then present the methodology of this evaluation, followed by a discussion of its quantita- tive results. Finally, I compare this evaluation with other empirical evaluations in text generation and conclude by discussing future directions. 2 An overview of STREAK The project STREAK was initially motivated by an- alyzing a corpus of newswire summaries written by professional sportswriters 2. This analysis revealed four characteristics of summaries that challenge the capabilities of previous text generators: concise lin- guistic forms, complex sentences, optional and back- ground facts opportunistically slipped as modifiers of obligatory facts and high paraphrasing power. By greatly increasing the number of content planning and linguistic realization options that the genera- tor must consider, as well as the mutual constraints among them, these characteristics …

Content truncated to 500 words