Performance Confidence Estimation for Automatic Summarization

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 541–548, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Performance Confidence Estimation for Automatic Summarization Annie Louis University of Pennsylvania lannie@seas.upenn.edu Ani Nenkova University of Pennsylvania nenkova@seas.upenn.edu Abstract We address the task of automatically pre- dicting if summarization system perfor- mance will be good or bad based on fea- tures derived directly from either single- or multi-document inputs. Our labelled cor- pus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We ex- amine different formulations of the classi- fication task which considerably influence performance. The best results are 84% prediction accuracy for single- and 74% for multi-document summarization. 1 Introduction The input to a summarization system significantly affects the quality of the summary that can be pro- duced for it, by either a person or an automatic method. Some inputs are difficult and summaries produced by any approach will tend to be poor, while other inputs are easy and systems will ex- hibit good performance. User satisfaction with the summaries can be improved, for example by auto- matically flagging summaries for which a system expects to perform poorly. In such cases the user can ignore the summary and avoid the frustration of reading poor quality text. (Brandow et al., 1995) describes an intelligent summarizer system that could identify documents which would be difficult to summarize based on structural properties. Documents containing ques- tion/answer sessions, speeches, tables and embed- ded lists were identified based on patterns and these features were used to determine whether an acceptable summary can be produced. If not, the inputs were flagged as unsuitable for automatic summarization. In our work, we provide deeper insight into how other characteristics of the text itself and properties of document clusters can be used to identify difficult inputs. The task of predicting the confidence in system performance for a given input is in fact relevant not only for summarization, but in general for all ap- plications aimed at facilitating information access. In question answering for example, a system may be configured not to answer questions for which the confidence of producing a correct answer is low, and in this way increase the overall accuracy of the system whenever it does produce an answer (Brill et al., 2002; Dredze and Czuba, 2007). Similarly in machine translation, some sen- tences might contain difficult to translate phrases, that is, portions of the input are likely to lead to garbled output if automatic translation is at- tempted. Automatically identifying such phrases has the potential of improving MT as shown by an oracle study (Mohit and Hwa, 2007). …

Content truncated to 500 words