The Importance of Calibration for Estimating Proportions from Annotations

Proceedings of NAACL-HLT 2018, pages 1636–1646 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics The Importance of Calibration for Estimating Proportions from Annotations Dallas Card Machine Learning Department Carnegie Mellon University Pittsburgh, PA, 15213, USA dcard@cmu.edu Noah A. Smith Paul G. Allen School of CSE University of Washington Seattle, WA, 98195, USA nasmith@cs.washington.edu Abstract Estimating label proportions in a target cor- pus is a type of measurement that is useful for answering certain types of social-scientific questions. While past work has described a number of relevant approaches, nearly all are based on an assumption which we argue is invalid for many problems, particularly when dealing with human annotations. In this paper, we identify and differentiate between two rele- vant data generating scenarios (intrinsic vs. ex- trinsic labels), introduce a simple but novel method which emphasizes the importance of calibration, and then analyze and experimen- tally validate the appropriateness of various methods for each of the two scenarios. 1 Introduction A methodological tool often used in the social sci- ences and humanities (and practical settings like journalism) is content analysis – the manual cat- egorization of pieces of text into a set of cate- gories which have been developed to answer a sub- stantive research question (Krippendorff, 2012). Automated content analysis holds great promise for augmenting the efforts of human annotators (O’Connor et al., 2011; Grimmer and Stewart, 2013). While this task bears similarity to text cat- egorization problems such as sentiment analysis, the quantity of real interest is often the proportion of documents in a dataset that should receive each label (Hopkins and King, 2010). This paper tack- les the problem of estimating label proportions in a target corpus based on a small sample of human annotated data. As an example, consider the hypothetical ques- tion (not explored in this work) of whether hate speech is increasingly prevalent in social media posts in recent years. “Hate speech” is a difficult- to-define category only revealed (at least initially) through human judgments (Davidson et al., 2017). Note that the goal would not be to identify individ- ual instances, but rather to estimate a proportion, as a way of measuring the prevalence of a social phenomenon. Although we assume that trained annotators could recognize this phenomenon with some acceptable level of agreement, relying solely on manual annotation would restrict the number of messages that could be considered, and would limit the analysis to the messages available at the time of annotation.1 We thus treat proportion estimation as a mea- surement problem, and seek a way to train an in- strument from a limited number of human anno- tations to measure label proportions in an unanno- tated target corpus. This problem can be cast within a supervised learning framework, and past work has demon- strated that it is possible to improve upon a na¨ıve classification-based approach, even without access to any labeled data from the target corpus (For- man, 2005, 2008; Bella et al., 2010; Hopkins and King, 2010; …

Content truncated to 500 words