CLaC and CLaC-NB: Knowledge-based and corpus-based approaches to sentiment tagging

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 117–120, Prague, June 2007. c⃝2007 Association for Computational Linguistics CLaC and CLaC-NB: Knowledge-based and corpus-based approaches to sentiment tagging Alina Andreevskaia Concordia University 1455 de Maisonneuve Blvd. Montreal, Canada andreev@cs.concordia.ca Sabine Bergler Concordia University 1455 de Maisonneuve Blvd. Montreal, Canada bergler@cs.concordia.ca Abstract For the Affective Text task at Semeval- 1/Senseval-4, the CLaC team compared a knowledge-based, domain-independent ap- proach and a standard, statistical machine learning approach to ternary sentiment an- notation of news headlines. In this paper we describe the two systems submitted to the competition and evaluate their results. We show that the knowledge-based unsu- pervised method achieves high accuracy and precision but low recall, while supervised statistical approach trained on small amount of in-domain data provides relatively high recall at the cost of low precision. 1 Introduction Sentiment tagging of short text spans — sentences, headlines, or clauses — poses considerable chal- lenges for automatic systems due to the scarcity of sentiment clues in these units: sometimes, the deci- sion about the text span sentiment has to be based on just a single sentiment clue and the cost of every error is high. This is particularly true for headlines, which are typically very short. Therefore, an ideal system for sentiment tagging of headlines has to use a large set of features with dependable sentiment an- notations and to be able to reliably deduce the senti- ment of the headline from the sentiment of its com- ponents. The valence labeling subtask of the Affective Text task requires ternary — positive vs. negative vs. neutral — classification of headlines. While such categorization at the sentence level remains rela- tively unexplored1, the two related sentence-level, binary classification tasks — positive vs. negative and subjective vs. objective — have attracted con- siderable attention in the recent years (Hu and Liu, 2004; Kim and Hovy, 2005; Riloff et al., 2006; Tur- ney and Littman, 2003; Yu and Hatzivassiloglou, 2003). Unsupervised knowledge-based methods are the preferred approach to classification of sentences into positive and negative, mostly due to the lack of adequate amounts of labeled training data (Gamon and Aue, 2005). These approaches rely on presence and scores of sentiment-bearing words that have been acquired from dictionaries (Kim and Hovy, 2005) or corpora (Yu and Hatzivassiloglou, 2003). Their accuracy on news sentences is between 65 and 68%. Sentence-level subjectivity detection, where train- ing data is easier to obtain than for positive vs. neg- ative classification, has been successfully performed using supervised statistical methods alone (Pang and Lee, 2004) or in combination with a knowledge- based approach (Riloff et al., 2006). Since the extant literature does not provide clear evidence for the choice between supervised machine learning methods and unsupervised knowledge- based approaches for the task of ternary sentiment classification of sentences or headlines, we devel- oped two systems for the Affective Text task at SemEval-2007. The first system (CLaC) relies on the knowledge-rich approach that takes into consid- 1To our knowledge, the only …

Content truncated to 500 words