Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7053–7065 August 1–6, 2021. ©2021 Association for Computational Linguistics 7053 Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability Ka Wong Google Research danicky@gmail.com Praveen Paritosh Google Research pkp@google.com Lora Aroyo Google Research l.m.aroyo@gmail.com Abstract When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012). Metrics such as Krippendorff’s alpha or Cohen’s kappa are typically required to be above a threshold of 0.6 (Landis and Koch, 1977). These abso- lute thresholds are unreasonable for crowd- sourced data from annotators with high cul- tural and training variances, especially on sub- jective topics. We present a new alternative to interpreting IRR that is more empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication relia- bility (xRR) measure based on Cohen’s (1960) kappa. We call this approach the xRR frame- work. We opensource a replication dataset of 4 million human judgements of facial expres- sions and analyze it with the proposed frame- work. We argue this framework can be used to measure the quality of crowdsourced datasets. 1 Introduction Much content analysis and linguistics research is based on data generated by human beings (hence- forth, annotators or raters) asked to make some kind of judgment. These judgments involve systematic interpretation of textual, visual, or audible mat- ter (e.g. newspaper articles, television programs, advertisements, public speeches, and other multi- modal data). When relying on human observers, researchers must worry about the quality of the data — specifically, their reliability (Krippendorff, 2004). Are the annotations collected reproducible, or are they the result of human idiosyncrasies? Respectable scholarly journals typically require reporting quantitative evidence for the inter-rater reliability (IRR) of the data (Hallgren, 2012). Co- hen’s kappa (Cohen, 1960) or Krippendorff’s alpha (Hayes and Krippendorff, 2007) is expected to be Figure 1: Agreement measures for categorical data (Landis and Koch, 1977) above a certain threshold to be worthy of publica- tion, typically 0.6 (Landis and Koch, 1977). Simi- lar IRR requirements for human annotations data have been followed across many fields. In this pa- per we refer to this absolute interpretation of IRR as the Landis-Koch approach (Fig. 1). This approach has been foundational in guid- ing the development of widely used and shared datasets and resources. Meanwhile, the landscape of human annotations collection has witnessed a tectonic shift in recent years. Driven by the data- hungry success of machine learning (LeCun et al., 2015; Schaekermann et al., 2020), there has been an explosive growth in the use of crowdsourcing for building datasets and benchmarks (Snow et al., 2008; Kochhar et al., 2010). We identify three paradigm shifts in the scope of and methodologies for data collection that make the Landis-Koch ap- proach not as useful in today’s settings. A rise in annotator diversity …

Content truncated to 500 words