Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5558–5570 August 1–6, 2021. ©2021 Association for Computational Linguistics 5558 Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition Xin Zhang1, Guangwei Xu, Yueheng Sun2, Meishan Zhang 1∗, Pengjun Xie 1School of New Media and Communication, Tianjin University, China 2College of Intelligence and Computing, Tianjin University, China {hsinz,yhs,zhangmeishan}@tju.edu.cn {ahxgwOnePiece,xpjandy}@gmail.com Abstract Crowdsourcing is regarded as one prospective solution for effective supervised learning, aim- ing to build large-scale annotated training data by crowd workers. Previous studies focus on reducing the influences from the noises of the crowdsourced annotations for supervised mod- els. We take a different point in this work, re- garding all crowdsourced annotations as gold- standard with respect to the individual annota- tors. In this way, we find that crowdsourcing could be highly similar to domain adaptation, and then the recent advances of cross-domain methods can be almost directly applied to crowdsourcing. Here we take named entity recognition (NER) as a study case, suggest- ing an annotator-aware representation learn- ing model that inspired by the domain adap- tation methods which attempt to capture ef- fective domain-aware features. We investigate both unsupervised and supervised crowdsourc- ing learning, assuming that no or only small- scale expert annotations are available. Exper- imental results on a benchmark crowdsourced NER dataset show that our method is highly ef- fective, leading to a new state-of-the-art perfor- mance. In addition, under the supervised set- ting, we can achieve impressive performance gains with only a very small scale of expert an- notations. 1 Introduction Crowdsourcing has gained a growing interest in the natural language processing (NLP) community, which helps hard NLP tasks such as named entity recognition (Finin et al., 2010; Derczynski et al., 2016), part-of-speech tagging (Hovy et al., 2014), relation extraction (Abad et al., 2017), translation (Zaidan and Callison-Burch, 2011), argument re- trieval (Mayhew et al., 2020), and others (Snow et al., 2008; Callison-Burch and Dredze, 2010) to ∗Corresponding author. text Andrea Ferrigato sprinted to his World Cup win A-1 B-PER I-PER O O O B-ORG I-ORG O A-2 O B-PER O O O B-MISC I-MISC O A-3 B-PER I-PER O O O B-ORG I-ORG O EXP B-PER I-PER O O O B-MISC I-MISC O Figure 1: A NER example with crowdsourced labels, A and EXP denote annotator and expert, respectively. collect a large scale dataset for supervised model training. In contrast to the gold-standard annota- tions labeled by experts, the crowdsourced annota- tions can be constructed quickly at a low cost with masses of crowd annotators (Snow et al., 2008; Nye et al., 2018). However, these annotations are rela- tively lower-quality with much-unexpected noise since the crowd annotators are not professional enough, which can make errors in complex and ambiguous contexts (Sheng et al., 2008). Previous crowdsourcing learning models strug- gle to reduce the influences of noises of the crowd- sourced annotations (Hsueh et al., 2009; Raykar and Yu, …

Content truncated to 500 words