Polarity Lexicon Building: to what Extent Is the Manual Effort Worth?

Polarity Lexicon Building: to what Extent Is the Manual Effort Worth? I˜naki San Vicente, Xabier Saralegi Elhuyar Fundazioa Osinalde Industrialdea 3, 20170 Usurbil {i.sanvicente,x.saralegi}@elhuyar.com Abstract Polarity lexicons are a basic resource for analyzing the sentiments and opinions expressed in texts in an automated way. This paper explores three methods to construct polarity lexicons: translating existing lexicons from other languages, extracting polarity lexicons from corpora, and annotating sentiments Lexical Knowledge Bases. Each of these methods require a different degree of human effort. We evaluate how much manual effort is needed and to what extent that effort pays in terms of performance improvement. Experiment setup includes generating lexicons for Basque, and evaluating them against gold standard datasets in different domains. Results show that extracting polarity lexicons from corpora is the best solution for achieving a good performance with reasonable human effort. Keywords: Sentiment Analysis, Polarity Lexicons, Evaluation 1. Introduction Research effort on the sentiment analysis field has seen exponentially increased in the last years, due to its applicability in areas such as VTIC (Technological Surveillance / Competitive Intelligence), marketing or reputation management. One of the main resources of sentiment analysis systems are the polarity lexicons, list of words with prior polarities. Much research has been done on methods for building such methods automatically due to the high cost of manually created lexicons. Then again, automatic methods often produce noisy resources. Very little work has been done on polarity lexicons for Basque, as is the case for other less resourced languages. Thus, when facing the task of creating such a resource, the doubt arised. Is it worth to make a great manual annotation effort? How much is the gain we obtain by manually annotating polarity words over automatically built polarity lexicons? This paper compares three strategies for building a polarity lexicon for a less-resourced language. We assumed that for languages of this type the availability of parallel corpora, MT systems and polarity-annotated data is very limited, and we avoided using such resources. We measured the time cost of the manual effort and the gain it brings in terms of accuracy in an extrinsic evaluation. This experiment was carried out for Basque. 2. State of the Art Polarity lexicons are key resource on sentiment analysis systems. We can group the methods for polarity lexicon building proposed in the literature into three main approaches: manually constructed lexicons (Stone et al., 1966), corpus-based methods (Hatzivassiloglou and McKeown, 1997; Mihalcea et al., 2007) and methods that rely on Lexical Knowledge Bases (LKB) (Kamps et al., 2004; Liu and Singh, 2004; Kim and Hovy, 2004) For major languages there are well known manually constructed lexicons, such as, General Inquirer(Stone et al., 1966), OpinionFinder(Wilson et al., 2005), or SO-CAL(Taboada et al., 2011). Due to the fact that a great human effort is needed to build such resources, some of them are semi-automatically constructed, and manually corrected afterwards. In this line of work, some researchers explore the possibility of using resources already existing in another language (e.g., lexicons, and/or annotated …

Content truncated to 500 words