Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction

Proceedings of NAACL-HLT 2019, pages 1471–1480 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1471 Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction Yinfei Yang Google AI yinfeiy@google.com Oshin Agarwal University of Pennsylvania oagarwal@seas.upenn.edu Chris Tar Google AI ctar@google.com Byron C. Wallace Northeastern University b.wallace@northeastern.edu Ani Nenkova University of Pennsylvania nenkova@seas.upenn.edu Abstract Modern NLP systems require high-quality an- notated data. In specialized domains, expert annotations may be prohibitively expensive. An alternative is to rely on crowdsourcing to reduce costs at the risk of introducing noise. In this paper we demonstrate that directly model- ing instance difficulty can be used to improve model performance, and to route instances to appropriate annotators. Our difficulty predic- tion model combines two learned representa- tions: a ‘universal’ encoder trained on out-of- domain data, and a task-specific encoder. Ex- periments on a complex biomedical informa- tion extraction task using expert and lay anno- tators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) us- ing difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing. Our experiments confirm the expec- tation that for specialized tasks expert annota- tions are higher quality than crowd labels, and hence preferable to obtain if practical. More- over, augmenting small amounts of expert data with a larger set of lay annotations leads to fur- ther improvements in model performance. 1 Introduction Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges. Ex- perts with the requisite domain knowledge to per- form high-quality annotation tend to be expen- sive, while lay annotators may not have the nec- essary knowledge to provide high-quality annota- tions. A practical approach for collecting a suf- ficiently large corpus would be to use crowd- sourcing platforms like Amazon Mechanical Turk (MTurk). However, crowd workers in general are likely to provide noisy annotations (Abad and Moschitti, 2016; Plank et al., 2014; Alonso et al., 2015), an issue exacerbated by the technical na- ture of specialized content. Some of this noise may reflect worker quality and can be modeled (Abad and Moschitti, 2016; Plank et al., 2014; Cohn and Specia, 2013; Nguyen et al., 2017), but for some instances lay people may simply lack the domain knowledge to provide useful annotation. In this paper we report experiments on the EBM-NLP corpus comprising crowdsourced an- notations of medical literature (Nye et al., 2018). We operationalize the concept of annotation dif- ficulty and show how it can be exploited during training to improve information extraction mod- els. We then obtain expert annotations for the ab- stracts predicted to be most difficult, as well as for a similar number of randomly selected abstracts. The annotation of highly specialized data and the use of lay and expert annotators allow us to exam- ine the following …

Content truncated to 500 words