MPQA 3.0: An Entity/Event-Level Sentiment Corpus

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1323–1328, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics MPQA 3.0: An Entity/Event-Level Sentiment Corpus Lingjia Deng Intelligent Systems Program University of Pittsburgh lid29@pitt.edu Janyce Wiebe Intelligent Systems Program Department of Computer Science University of Pittsburgh wiebe@cs.pitt.edu Abstract This paper presents an annotation scheme for adding entity and event target annotations to the MPQA corpus, a rich span-annotated opin- ion corpus. The new corpus promises to be a valuable new resource for developing sys- tems for entity/event-level sentiment analysis. Such systems, in turn, would be valuable in NLP applications such as Automatic Question Answering. We introduce the idea of entity and event targets (eTargets), describe the an- notation scheme, and present the results of an agreement study. 1 Introduction Much work in sentiment analysis and opinion min- ing is at the document level (Pang et al., 2002; Turney, 2002). There is increasing interest in more fine-grained levels - sentence-level (Yu and Hatzivassiloglou, 2003; McDonald et al., 2007), phrase-level (Choi and Cardie, 2008), aspect-level (Hu and Liu, 2004; Titov and McDonald, 2008), etc. We specifically address sentiments toward en- tities and events (i.e., eTargets) expressed in data such as blogs, newswire, and editorials. A system that could recognize sentiments toward entities and events would be valuable in an application such as Automatic Question Answering, to support answer- ing questions such as “Toward whom/what is X neg- ative/positive?” “Who is negative/positive toward X?” (Stoyanov et al., 2005). Or, to augment an automatic wikification system (Ratinov et al., 2011) – in addition to relationships such as spouse and parents, the system could include information about whom or what the subject supports or opposes. A re- cent NIST evaluation – The Knowledge Base Popu- lation (KBP) Sentiment track1 — aims at using cor- pora to collect information regarding sentiments ex- pressed toward or by named entities. Annotated corpora of reviews (e.g., (Hu and Liu, 2004; Titov and McDonald, 2008)), widely used in NLP, often include target annotations. Such targets are often aspects or features of products or services, and as such are somewhat limited.2 Recently, to create the Sentiment Treebank (Socher et al., 2013), researchers crowdsourced an- notations of movie review data and then overlaid the annotations onto syntax trees. Thus, the tar- gets are not limited to aspects of products/services. However, annotators were asked to annotate small and then increasingly larger segments of the sen- tence. Thus, the annotations are mixed in the de- gree to which context was considered when making the judgements. Previously, we (Deng et al., 2013) annotated a corpus of non-review data with senti- ments toward entities, but only for those that partic- ipate in certain types of events. In all of the above corpora, the only sentiments considered are those of the writer, excluding sentiments attributed to other entities. The MPQA opinion annotated corpus (Wiebe et al., 2005; Wilson, 2007) is entirely span-based, and contains no eTarget …

Content truncated to 500 words