All-Topology, Semi-Abstract Syntactic Features for Text Categorization

All-Topology, Semi-Abstract Syntactic Features for Text Categorisation Ari Chanen School of Information Technologies University of Sydney Sydney, Australia, 2006 ari@it.usyd.edu.au Jon Patrick School of Information Technologies University of Sydney Sydney, Australia, 2006 jonpat@it.usyd.edu.au Abstract Good performance on Text Classifica- tion (TC) tasks depends on effective and statistically significant features. Typ- ically, the simple bag-of-words repre- sentation is widely used because uni- gram counts are more likely to be signif- icant compared to more compound fea- tures. This research explores the idea that the major cause of poor perfor- mance of some complex features is spar- sity. Syntactic features are usually com- plex being made up of both lexical and syntactic information. This paper intro- duces the use of a class of automatically extractable, syntactic features to the TC task. These features are based on sub- trees of parse trees. As such, a large number of these features are generated. Our results suggest that generating a di- verse set of these features may help in increasing performance. Partial abstrac- tion of the features also seems to boost performance by counteracting sparsity. We will show that various subsets of our syntactic features do outperform the bag-of-words representation alone. 1 Introduction One of the keys to obtaining good performance out of an automatic TC system is choosing appropri- ate types of features. Humans can often do better than automatic classification systems on difficult- to-categorise corpora, probably by using the hu- man ability to extract the meaning of documents from the document words. Many researchers have tried to use features that are closer to a more semantic representation of documents. Unfortu- nately, many of these attempts do not succeed in doing better than bag-of-words e.g. (Moschitti and Basili, 2004). A major problem seems to be that the more complex a feature type is, the less frequently it occurs. A complex feature’s occu- rance rate in a training set will often not be signif- icant in an unseen test set. A simple example is how a purely bigram representation of documents is expected to do worse than a representation of the same documents using unigrams, as we will demonstrate below in the Experiments section. Attempting to use syntactic features can be ap- pealing because the syntax of document sentences holds clues into how an author encodes the seman- tics into the sequence of document words. Syntax can be thought of as the middle ground between the lexical and semantic levels of representation. An apt quote on this subject is “...the aim of syn- tactical representation is to constitute a bridgehead to semantics”(Schneider, 1998). The syntax of a document is usually represented in a hyper-linear representation such as trees. Syntactic trees are rich in information where the nodes encode in- formation about sentence terms and the links en- code information about the relationships between terms. It is tempting to construct complex features by extracting them from parse trees. However, there is always the problem of feature sparsity. The right type of complex, syntactic feature may be …

Content truncated to 500 words