Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization Regina Barzilay Computer Science and AI Lab MIT regina@csail.mit.edu Lillian Lee Department of Computer Science Cornell University llee@cs.cornell.edu Abstract We consider the problem of modeling the con- tent structure of texts within a specific do- main, in terms of the topics the texts address and the order in which these topics appear. We first present an effective knowledge-lean method for learning content models from un- annotated documents, utilizing a novel adap- tation of algorithms for Hidden Markov Mod- els. We then apply our method to two com- plementary tasks: information ordering and ex- tractive summarization. Our experiments show that incorporating content models in these ap- plications yields substantial improvement over previously-proposed methods. 1 Introduction The development and application of computational mod- els of text structure is a central concern in natural lan- guage processing. Document-level analysis of text struc- ture is an important instance of such work. Previous research has sought to characterize texts in terms of domain-independent rhetorical elements, such as schema items (McKeown, 1985) or rhetorical relations (Mann and Thompson, 1988; Marcu, 1997). The focus of our work, however, is on an equally fundamental but domain- dependent dimension of the structure of text: content. Our use of the term “content” corresponds roughly to the notions of topic and topic change. We desire models that can specify, for example, that articles about earthquakes typically contain information about quake strength, location, and casualties, and that descriptions of casualties usually precede those of rescue efforts. But rather than manually determine the topics for a given domain, we take a distributional view, learning them directly from un-annotated texts via analysis of word distribution patterns. This idea dates back at least to Harris (1982), who claimed that “various types of [word] recurrence patterns seem to characterize various types of discourse”. Advantages of a distributional perspective in- clude both drastic reduction in human effort and recogni- tion of “topics” that might not occur to a human expert and yet, when explicitly modeled, aid in applications. Of course, the success of the distributional approach depends on the existence of recurrent patterns. In arbi- trary document collections, such patterns might be too variable to be easily detected by statistical means. How- ever, research has shown that texts from the same domain tend to exhibit high similarity (Wray, 2002). Cognitive psychologists have long posited that this similarity is not accidental, arguing that formulaic text structure facilitates readers’ comprehension and recall (Bartlett, 1932).1 In this paper, we investigate the utility of domain- specific content models for representing topics and topic shifts. Content models are Hidden Markov Models (HMMs) wherein states correspond to types of information characteristic to the domain of in- terest (e.g., earthquake magnitude or previous earth- quake occurrences), and state transitions capture possible information-presentation orderings within that domain. We first describe an efficient, knowledge-lean method for learning both a set of topics and the relations be- tween topics directly from un-annotated documents. Our technique …

Content truncated to 500 words