CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1913–1922 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles Chenguang Wang† , Alan Akbik‡∗, Laura Chiticariu† , Yunyao Li† , Fei Xia§ , Anbang Xu† †IBM Research - Almaden ‡Zalando Research, Berlin §Department of Linguistics, University of Washington chenguang.wang@ibm.com, alan.akbik@zalando.de {chiti, yunyaoli, anbangxu}@us.ibm.com, fxia@uw.edu Abstract Crowdsourcing has proven to be an effec- tive method for generating labeled data for a range of NLP tasks. However, multiple recent attempts of using crowdsourcing to generate gold-labeled training data for se- mantic role labeling (SRL) reported only modest results, indicating that SRL is per- haps too difficult a task to be effectively crowdsourced. In this paper, we postu- late that while producing SRL annotation does require expert involvement in gen- eral, a large subset of SRL labeling tasks is in fact appropriate for the crowd. We present a novel workflow in which we em- ploy a classifier to identify difficult an- notation tasks and route each task either to experts or crowd workers according to their difficulties. Our experimental eval- uation shows that the proposed approach reduces the workload for experts by over two-thirds, and thus significantly reduces the cost of producing SRL annotation at little loss in quality. 1 Introduction Semantic role labeling (SRL) is the task of label- ing the predicate-argument structures of sentences with semantic frames and their roles (Baker et al., 1998; Palmer et al., 2005). It has been found useful for a wide variety of NLP tasks such as question-answering (Shen and Lapata, 2007), in- formation extraction (Fader et al., 2011) and ma- chine translation (Lo et al., 2013). A major bot- tleneck impeding the wide adoption of SRL is the need for large amounts of labeled training data to ∗The work was done while the author was at IBM Re- search - Almaden. capture broad-coverage semantics. Such data re- quires trained experts and is highly costly to pro- duce (Hovy et al., 2006). Crowdsourcing SRL Crowdsourcing has shown its effectiveness to generate labeled data for a range of NLP tasks (Snow et al., 2008; Hong and Baker, 2011; Franklin et al., 2011). A core ad- vantage of crowdsourcing is that it allows the an- notation workload to be scaled out among large numbers of inexpensive crowd workers. Not sur- prisingly, a number of recent SRL works have also attempted to leverage crowdsourcing to gen- erate labeled training data for SRL and investi- gated a variety of ways of formulating crowd- sourcing tasks (Fossati et al., 2013; Pavlick et al., 2015; Akbik et al., 2016). All have found that crowd feedback generally suffers from low inter- annotator agreement scores and often produces in- correct labels. These results seem to indicate that, regardless of the design of the task, SRL is simply too difficult to be effectively crowdsourced. Proposed Approach We observe that there are significant differences in difficulties among SRL annotation tasks, depending on factors …

Content truncated to 500 words