Supervised Ranking in Open-Domain Text Summarization

Supervised Ranking in Open-Domain Text Summarization Tadashi Nomoto National Institute of Japanese Literature 1-16-10 Yutaka Shinagawa Tokyo 142-8585, Japan nomoto@nijl.ac.jp Yuji Matsumoto Nara Institute of Science and Technology 8916-5 Takayama Ikoma Nara 630-0101, Japan matsu@is.aist-nara.ac.jp Abstract The paper proposes and empirically moti- vates an integration of supervised learning with unsupervised learning to deal with human biases in summarization. In par- ticular, we explore the use of probabilistic decision tree within the clustering frame- work to account for the variation as well as regularity in human created summaries. The corpus of human created extracts is created from a newspaper corpus and used as a test set. We build probabilistic de- cision trees of different flavors and in- tegrate each of them with the clustering framework. Experiments with the cor- pus demonstrate that the mixture of the two paradigms generally gives a signif- icant boost in performance compared to cases where either of the two is considered alone. 1 Introduction Nomoto and Matsumoto (2001b) have recently made an interesting observation that an unsu- pervised method based on clustering sometimes better approximates human created extracts than a supervised approach. That appears somewhat con- tradictory given that a supervised approach should be able to exploit human supplied information about which sentence to include in an extract and which not to, whereas an unsupervised approach blindly chooses sentences according to some selection scheme. An interesting question is, why this should be the case. The reason may have to do with the variation in human judgments on sentence selection for a sum- mary. In a study to be described later, we asked stu- dents to select 10% of a text which they find most important for making a summary. If they agree per- fectly on their judgments, then we will have only 10% of a text selected as most important. However, what we found was that about half of a text were marked as important, indicating that judgments can vary widely among humans. Curiously, however, Nomoto and Matsumoto (2001a) also found that a supervised system fares much better when tested on data exhibiting high agreement among humans than an unsupervised sys- tem. Their finding suggests that there are indeed some regularities (or biases) to be found. So we might conclude that there are two aspects to human judgments in summarization; they can vary but may exhibit some biases which could be usefully exploited. The issue is then how we might model them in some coherent framework. The goal of the paper is to explore a possible in- tegration of supervised and unsupervised paradigms as a way of responding to the issue. Taking a de- cision tree and clustering as representing the respec- tive paradigm, we will show how coupling them pro- vides a summarizer that better approximates human judgments than either of the two considered alone. To our knowledge, none of the prior work on sum- marization (e.g., Kupiec et al. (1995)) explicitly ad- dressed the issue of the variability inherent in human judgments in summarization …

Content truncated to 500 words