Automatic Detection of Text Genre

Automatic Detection of Text Genre Brett Kessler Geoffrey Nunberg Hinrich Schfitze Xerox Palo Alto Research Center 3333 Coyote Hill Road Palo Alto CA 94304 USA Department of Linguistics Stanford University Stanford CA 94305-2150 USA emaih {bkessler,nunberg,schuetze}~parc.xerox.com URL: ftp://parcftp.xerox.com/pub/qca/papers/genre Abstract As the text databases available to users be- come larger and more heterogeneous, genre becomes increasingly important for com- putational linguistics as a complement to topical and structural principles of classifi- cation. We propose a theory of genres as bundles of facets, which correlate with var- ious surface cues, and argue that genre de- tection based on surface cues is as success- ful as detection based on deeper structural properties. 1 Introduction Computational linguists have been concerned for the most part with two aspects of texts: their structure and their content. That is. we consider texts on the one hand as formal objects, and on the other as symbols with semantic or referential values. In this paper we want to consider texts from the point of view of genre: that is. according to the various functional roles they play. Genre is necessarily a heterogeneous classificatory principle, which is based among other things on the way a text was created, the way it is distributed, the register of language it uses, and the kind of au- dience it is addressed to. For all its complexity, this attribute can be extremely important for many of the core problems that computational linguists are concerned with. Parsing accuracy could be increased by taking genre into account (for example, certain object-less constructions occur only in recipes in En- glish). Similarly for POS-tagging (the frequency of uses of trend as a verb in the Journal of Commerce is 35 times higher than in Sociological Abstracts). In word-sense disambiguation, many senses are largely restricted to texts of a particular style, such as col- loquial or formal (for example the word pretty is far more likely to have the meaning "rather" in informal genres than in formal ones). In information retrieval, genre classification could enable users to sort search results according to their immediate interests. Peo- ple who go into a bookstore or library are not usually looking simply for information about a particular topic, but rather have requirements of genre as well: they are looking for scholarly articles about hypno- tism, novels about the French Revolution, editorials about the supercollider, and so forth. If genre classification is so useful, why hasn't it fig- ured much in computational linguistics before now? One important reason is that, up to now, the digi- tized corpora and collections which are the subject of much CL research have been for the most part generically homogeneous (i.e., collections of scientific abstracts or newspaper articles, encyclopedias, and so on), so that the problem of genre identification could be set aside. To a large extent, the problems of genre classification don't become salient until we are confronted with large and heterogeneous search domains like the World-Wide Web. Another reason for the neglect of genre, though, is …

Content truncated to 500 words