Selecting Systemic Features for Text Classification

Selecting Systemic Features for Text Classification Casey Whitelaw and Jon Patrick Language Technology Research Group Capital Markets Co-operative Research Centre School of Information Technologies University of Sydney {casey, jonpat}@it.usyd.edu.au Abstract Systemic features use linguistically- derived language models as a basis for text classification. The graph structure of these models allows for feature repre- sentations not available with traditional bag-of-words approaches. This paper explores the set of possible represen- tations, and proposes feature selection methods that aim to produce the most compact and effective set of attributes for a given classification problem. We show that small sets of systemic fea- tures can outperform larger sets of word- based features in the task of identifying financial scam documents. 1 Introduction Text classification is among the most widespread applications of computational linguistics. The range of services that offer text classification con- tinue to grow, and include such mainstream appli- cations as email and web content filtering. The classification of documents by machine learning techniques requires a representation of each doc- ument as a set of features; almost without excep- tion, these features are based on the presence, ab- sence, or frequency of words in the text. This ‘bag- of-words’ model is popular both due to its ease of implementation, and its excellent performance on many tasks. Topic-based classification, such as newswire or newsgroup tasks, is well-suited to this automated keyword-spotting approach. These are cases in which the presence of a topic-related word such as ‘wheat’ is a very strong indicator of a document’s class. The bag-of-words model makes large simplify- ing assumptions about a document. It assumes that there is no textual structure; no ordering of paragraphs in the text, sentences in a paragraph, clauses in a sentence, or words in a clause. In addition, it assumes that the occurrence of each word is independent of each other word. These assumptions, in providing a much simpler picture of the document, destroy much of the text’s mean- ing. Work has been done to restore this infor- mation using semantic resources such as Word- Net (Scott and Matwin, 1998) or using syntactic information (Carr and Estival, 2002). There is also growing interest in classifying texts on non- denotational meaning, such as writing style, au- thorship identification (van Halteren, 2004) and sentiment analysis (Pang and Lee, 2004). These new areas highlight the properties of a document that are currently slipping through the cracks. This paper takes another approach to providing a better representation than bag-of-words. In line with Systemic Functional Linguistic theory, the words of a text are treated as evidence of semantic choices being made by the author. These choices form systems, and each document is modelled as the set of choices it makes within these systems. This knowledge of the semantic relationships be- tween features allows for more sophisticated rep- resentations that more accurately capture charac- teristic linguistic differences. In Section 2 we enu- merate these representations and discuss how their semantics differ. Section 3 describes the Scam- seek project and the use of …

Content truncated to 500 words