Fine-Grained Genre Classification Using Structural Learning Algorithms

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 749–759, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Fine-grained Genre Classification using Structural Learning Algorithms Zhili Wu Centre for Translation Studies University of Leeds, UK z.wu@leeds.ac.uk Katja Markert School of Computing University of Leeds, UK scskm@leeds.ac.uk Serge Sharoff Centre for Translation Studies University of Leeds, UK s.sharoff@leeds.ac.uk Abstract Prior use of machine learning in genre classification used a list of labels as clas- sification categories. However, genre classes are often organised into hierar- chies, e.g., covering the subgenres of fic- tion. In this paper we present a method of using the hierarchy of labels to improve the classification accuracy. As a testbed for this approach we use the Brown Cor- pus as well as a range of other corpora, in- cluding the BNC, HGC and Syracuse. The results are not encouraging: apart from the Brown corpus, the improvements of our structural classifier over the flat one are not statistically significant. We discuss the relation between structural learning per- formance and the visual and distributional balance of the label hierarchy, suggesting that only balanced hierarchies might profit from structural learning. 1 Introduction Automatic genre identification (AGI) can be traced to the mid-1990s (Karlgren and Cutting, 1994; Kessler et al., 1997), but this research be- came much more active in recent years, partly be- cause of the explosive growth of the Web, and partly because of the importance of making genre distinctions in NLP applications. In Information Retrieval, given the large number of web pages on any given topic, it is often difficult for the users to find relevant pages that are in the right genre (Vidulin et al., 2007). As for other applications, the accuracy of many tasks, such as machine trans- lation, POS tagging (Giesbrecht and Evert, 2009) or identification of discourse relations (Webber, 2009) relies of defining the language model suit- able for the genre of a given text. For example, the accuracy of POS tagging reaching 96.9% on newspaper texts drops down to 85.7% on forums (Giesbrecht and Evert, 2009), i.e., every seventh word in forums is tagged incorrectly. This interest in genres resulted in a prolifer- ation of studies on corpus development of web genres and comparison of methods for AGI. The two corpora commonly used for this task are KI- 04 (Meyer zu Eissen and Stein, 2004) and San- tinis (Santini, 2007). The best results reported for these corpora (with 10-fold cross-validation) reach 84.1% on KI-04 and 96.5% accuracy on Santinis (Kanaris and Stamatatos, 2009). In our research (Sharoff et al., 2010) we produced even better re- sults on these two benchmarks (85.8% and 97.1%, respectively). However, this impressive accuracy is not realistic in vivo, i.e., in classifying web pages retrieved as a result of actual queries. One reason comes from the limited number of genres present in these two collections (eight genres in KI-04 and seven in Santinis). As an example, only front pages of online newspapers are listed …

Content truncated to 500 words