Toward Annotator Group Bias in Crowdsourcing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1797 - 1806 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Toward Annotator Group Bias in Crowdsourcing Haochen Liu1, Joseph Thekinen2∗, Sinem Mollaoglu1, Da Tang3, Ji Yang3, Youlong Cheng3, Hui Liu1, Jiliang Tang1 1 Michigan State University, East Lansing, MI, USA 2 University of Calgary, Calgary, AB, Canada 3 ByteDance Inc., Mountain View, CA, USA liuhaoc1@msu.edu; joseph.thekinen@ucalgary.ca; sinemm@msu.edu; {da.tang,ji.yang,youlong.cheng}@bytedance.com; {liuhui7,tangjili}@msu.edu Abstract Crowdsourcing has emerged as a popular ap- proach for collecting annotated data to train supervised machine learning models. However, annotator bias can lead to defective annotations. Though there are a few works investigating in- dividual annotator bias, the group effects in annotators are largely overlooked. In this work, we reveal that annotators within the same de- mographic group tend to show consistent group bias in annotation tasks and thus we conduct an initial study on annotator group bias. We first empirically verify the existence of annotator group bias in various real-world crowdsourcing datasets. Then, we develop a novel probabilis- tic graphical framework GroupAnno to cap- ture annotator group bias with an extended Ex- pectation Maximization (EM) algorithm. We conduct experiments on both synthetic and real- world datasets. Experimental results demon- strate the effectiveness of our model in model- ing annotator group bias in label aggregation and model learning over competitive baselines. 1 Introduction The performance of supervised machine learning algorithms heavily relies on the quality of the anno- tated training data. Due to the heavy workload of annotation tasks, researchers and practitioners typi- cally take advantage of crowdsourcing platforms to obtain cost-effective annotation data (Snow et al., 2008; Buhrmester et al., 2016). However, the labels collected from multiple crowdsourcing annotators could be not consistent, since the expertise and reli- ability of the annotators are uncertain, and the task itself could be subjective and difficult. In recent years, a lot of efforts from the machine learning community have been conducted to mitigate the effect of these noisy crowdsourcing labels (Zheng et al., 2017). Various approaches have been pro- posed to model the quality (Liu et al., 2012; Aydin et al., 2014), confidence (Joglekar et al., 2013), ∗ Corresponding author. expertise (Ma et al., 2015; Zheng et al., 2016), re- liability (Li et al., 2019) of annotators; or model the difficulty of the tasks (Whitehill et al., 2009; Ma et al., 2015). With such information, we can infer the truth label from the noisy labels more ac- curately and correspondingly train a more desirable model. In terms of annotator modeling, existing studies mainly concentrated on factors like quality, confi- dence, expertise, etc., which could affect the anno- tation results. Besides, the bias held by the anno- tators can also lead to defective annotations (Sap et al., 2019), which is, however, rarely studied. In addition, studies in social science (Eagly, 2013) suggest that people from different demographic groups tend to apply different standards to eval- uate the same thing due to their different experi- ences, …

Content truncated to 500 words