Using Pivot-Based Paraphrasing and Sentiment Profiles to Improve a Subjectivity Lexicon for Essay Data

Transactions of the Association for Computational Linguistics, 1 (2013) 99–110. Action Editor: Chris Callison-Burch. Submitted 12/2012; Published 5/2013. c⃝2013 Association for Computational Linguistics. Using Pivot-Based Paraphrasing and Sentiment Profiles to Improve a Subjectivity Lexicon for Essay Data Beata Beigman Klebanov, Nitin Madnani, Jill Burstein Educational Testing Service 660 Rosedale Road, Princeton, NJ 08541, USA {bbeigmanklebanov,nmadnani,jburstein@ets.org} Abstract We demonstrate a method of improving a seed sentiment lexicon developed on essay data by using a pivot-based paraphrasing system for lexical expansion coupled with sentiment pro- file enrichment using crowdsourcing. Profile enrichment alone yields up to 15% improve- ment in the accuracy of the seed lexicon on 3- way sentence-level sentiment polarity classifi- cation of essay data. Using lexical expansion in addition to sentiment profiles provides a further 7% improvement in performance. Ad- ditional experiments show that the proposed method is also effective with other subjectivity lexicons and in a different domain of applica- tion (product reviews). 1 Introduction In almost any sub-field of computational linguistics, creation of working systems starts with an invest- ment in manually-generated or manually-annotated data for computational exploration. In subjectivity and sentiment analysis, annotation of training and testing data and construction of subjectivity lexicons have been the loci of costly labor investment. Many subjectivity lexicons are mentioned in the literature. The two large manually-built lexicons for English – the General Inquirer (Stone et al., 1966) and the lexicon provided with the Opinion- Finder distribution (Wiebe and Riloff, 2005) – are available for research and education only1 and un- der GNU GPL license that disallows their incor- poration into proprietary materials,2 respectively. 1http://www.wjh.harvard.edu/ inquirer/j1 1/manual/ 2http://www.gnu.org/copyleft/gpl.html Those wishing to integrate sentiment analysis into products, along with those studying subjectivity in languages other than English, or for specific do- mains such as finance, or for particular genres such as MySpace comments, reported construction of lexicons (Taboada et al., 2011; Loughran and McDonald, 2011; Thelwall et al., 2010; Rao and Ravichandran, 2009; Jijkoun and Hofmann, 2009; Pitel and Grefenstette, 2008; Mihalcea et al., 2007). In this paper, we address the step of expanding a small-scale, manually-built subjectivity lexicon (a seed lexicon, typically for a domain or language in question) into a much larger but noisier lexi- con using an automatic procedure. We present a novel expansion method using a state-of-the-art paraphrasing system. The expansion yields a 4-fold increase in lexicon size; yet, the expansion alone is insufficient in order to improve performance on sentence-level sentiment polarity classification. In this paper we test the following hypothesis. We suggest that the effectiveness of the expansion is hampered by (1) introduction of opposite-polarity items, such as introducing resolute as an expansion of forceful, or remarkable as an expansion of pecu- liar; (2) introduction of weakly polar, neutral, or am- biguous words as expansions of polar seed words, such as generating concern as an expansion of anx- iety or future as an expansion of aftermath;3 (3) in- ability to distinguish between stronger or clear-cut versus weaker or ambiguous sentiment and to make a differential use …

Content truncated to 500 words