Sentence and Expression Level Annotation of Opinions in User-Generated Discourse

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575–584, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Sentence and Expression Level Annotation of Opinions in User-Generated Discourse Cigdem Toprak and Niklas Jakob and Iryna Gurevych Ubiquitous Knowledge Processing Lab Computer Science Department, Technische Universit¨at Darmstadt, Hochschulstraße 10 D-64289 Darmstadt, Germany www.ukp.tu-darmstadt.de Abstract In this paper, we introduce a corpus of consumer reviews from the rateitall and the eopinions websites annotated with opinion-related information. We present a two-level annotation scheme. In the first stage, the reviews are analyzed at the sentence level for (i) relevancy to a given topic, and (ii) expressing an eval- uation about the topic. In the second stage, on-topic sentences containing eval- uations about the topic are further investi- gated at the expression level for pinpoint- ing the properties (semantic orientation, intensity), and the functional components of the evaluations (opinion terms, targets and holders). We discuss the annotation scheme, the inter-annotator agreement for different subtasks and our observations. 1 Introduction There has been a huge interest in the automatic identification and extraction of opinions from free text in recent years. Opinion mining spans a va- riety of subtasks including: creating opinion word lexicons (Esuli and Sebastiani, 2006; Ding et al., 2008), identifying opinion expressions (Riloff and Wiebe, 2003; Fahrni and Klenner, 2008), identi- fying polarities of opinions in context (Breck et al., 2007; Wilson et al., 2005), extracting opinion targets (Hu and Liu, 2004; Zhuang et al., 2006; Cheng and Xu, 2008) and opinion holders (Kim and Hovy, 2006; Choi et al., 2005). Data-driven approaches for extracting opinion expressions, their holders and targets require re- liably annotated data at the expression level. In previous research, expression level annotation of opinions was extensively investigated on newspa- per articles (Wiebe et al., 2005; Wilson and Wiebe, 2005; Wilson, 2008b) and on meeting dialogs (So- masundaran et al., 2008; Wilson, 2008a). Compared to the newspaper and meeting dialog genres, little corpus-based work has been carried out for interpreting the opinions and evaluations in user-generated discourse. Due to the high popular- ity of Web 2.0 communities1, the amount of user- generated discourse and the interest in the analysis of such discourse has increased over the last years. To the best of our knowledge, there are two cor- pora of user-generated discourse which are anno- tated for opinion related information at the expres- sion level: The corpus of Hu & Liu (2004) consists of customer reviews about consumer electronics, and the corpus of Zhuang et al. (2006) consists of movie reviews. Both corpora are tailored for ap- plication specific needs, therefore, do not contain certain related information explicitly annotated in the discourse, which we consider important (see Section 2). Furthermore, none of these works pro- vide inter-annotator agreement studies. Our goal is to create sentence and expression level annotated corpus of customer reviews which fulfills the following requirements: (1) It filters individual sentences regarding their topic rele- vancy and the existence of an opinion or …

Content truncated to 500 words