Development and Use of a Gold-Standard Data Set for Subjectivity Classifications

Development and Use of a Gold-Standard Data Set for Subjectivity Classifications Janyce M. Wiebet and Rebecca F. Bruce:[: and Thomas P. O'Harat tDepartment of Computer Science and Computing Research Laboratory New Mexico State University, Las Cruces, NM 88003 :~Department of Computer Science University of North Carolina at Asheville Asheville, NC 28804-8511 wiebe, tomohara@cs, nmsu. edu, bruce@cs, unca. edu Abstract This paper presents a case study of analyzing and improving intercoder reliability in discourse tagging using statistical techniques. Bias- corrected tags are formulated and successfully used to guide a revision of the coding manual and develop an automatic classifier. 1 Introduction This paper presents a case study of analyz- ing and improving intercoder reliability in dis- course tagging using the statistical techniques presented in (Bruce and Wiebe, 1998; Bruce and Wiebe, to appear). Our approach is data driven: we refine our understanding and pre- sentation of the classification scheme guided by the results of the intercoder analysis. We also present the results of a probabilistic classifier developed on the resulting annotations. Much research in discourse processing has focused on task-oriented and instructional di- alogs. The task addressed here comes to the fore in other genres, especially news reporting. The task is to distinguish sentences used to ob- jectively present factual information from sen- tences used to present opinions and evaluations. There are many applications for which this dis- tinction promises to be important, including text categorization and summarization. This research takes a large step toward developing a reliably annotated gold standard to support experimenting with such applications. This research is also a case study of ana- lyzing and improving manual tagging that is applicable to any tagging task. We perform a statistical analysis that provides information that complements the information provided by Cohen's Kappa (Cohen, 1960; Carletta, 1996). In particular, we analyze patterns of agreement to identify systematic disagreements that result from relative bias among judges, because they can potentially be corrected automatically. The corrected tags serve two purposes in this work. They are used to guide the revision of the cod- ing manual, resulting in improved Kappa scores, and they serve as a gold standard for developing a probabilistic classifier. Using bias-corrected tags as gold-standard tags is one way to define a single best tag when there are multiple judges who disagree. The coding manual and data from our exper- iments are available at: http://www.cs.nmsu.edu/~wiebe/projects. In the remainder of this paper, we describe the classification being performed (in section 2), the statistical tools used to analyze the data and produce the bias-corrected tags (in section 3), the case study of improving intercoder agree- ment (in section 4), and the results of the clas- sifter for automatic subjectivity tagging (in sec- tion 5). 2 The Subjective and Objective Categories We address evidentiality in text (Chafe, 1986), which concerns issues such as what is the source of information, and whether information is be- ing presented as fact or opinion. These ques- tions are particularly important in news report- ing, in which segments …

Content truncated to 500 words