Evaluating the Utility of Appraisal Hierarchies as a Method for Sentiment Classification

Proceedings of the Australasian Language Technology Workshop 2005, pages 134–142, Sydney, Australia, December 2005. Evaluating the Utility of Appraisal Hierarchies as a Method for Sentiment Classification Jeremy FLETCHER and Jon PATRICK Sydney Language Technology Research Group The University of Sydney Sydney, Australia, 2006 {jeremy, jonpat}@it.usyd.edu.au Abstract Recent studies of sentiment classification (determining whether a text is “positive” or “negative”) using Appraisal theory have provided mixed results. While some good results have been obtained, it is difficult to tell what aspects of Appraisal are particularly useful for this task. In this paper, we present a series of experiments to isolate features of Appraisal, in order to compare which parts aid the task of sentiment classification on movie reviews. We report results which on the surface challenge the utility of Appraisal Hierarchies for this task, when modelled using systemic features. However in the context of making a trade-off between coverage and scale of feature space, our results appear promising. We hence discuss the need for a balance between the size of a classifier’s structure and the overall accuracy. 1. Introduction Sentiment classification is a field of growing interest in the computational linguistics world, as researchers see the need for what has been termed non-topical text analysis. Sentiment classification deals with the problem of determining whether a document is positive or negative. This task has wide-ranging applications, notably market research, and customer feedback. This paper sets about to determine the usefulness of the linguistic theory of Appraisal for Sentiment Classification. Appraisal theory describes how opinion is expressed in text. Its description is in the form of system networks denoted by a taxonomy of expressions. In this work we rely on the description of these taxonomies in Martin and White’s The Language of Evaluation: Appraisal in English (2005), for both our linguistically guided hierarchies and realisations of features1. 1 In this study, we use only the ATTITUDE system from Martin and White’s Appraisal structure, and append a Figure 1 shows a visualisation of the system network we use from appraisal theory. Intuitively, it would seem that appraisal, if it could be modelled effectively using computational methods, would be a useful tool for sentiment analysis. A linguistic theory which gives us insight into the underlying construction of the opinion of the author of a piece of text should, in theory, allow us to compute such opinion more effectively. Previous work on using Appraisal for Sentiment Analysis, however, has been unconvincing and somewhat inconclusive. In this paper, we set about trying to isolate the areas of appraisal theory which are useful and applicable to sentiment analysis. Subsequently, we wish to determine where efforts in the automatic extraction of a document’s appraisal profile should be focussed. 2. Previous Work There has been some work done on the use of Appraisal for sentiment analysis, including the work of Taboada and Grieve (2004) in which different categories of product reviews were analysed for different types of Attitude (the three sub-systems being Affect, Judgment and Appreciation), using adjectives which had …

Content truncated to 500 words