Tell Me What You Read: Automatic Expertise-Based Annotator Assignment for Text Annotation in Expert Domains

Proceedings of Recent Advances in Natural Language Processing, pages 1575–1585 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_177 1575 Tell Me What You Read: Automatic Expertise-Based Annotator Assignment for Text Annotation in Expert Domains Hiyori Yoshikawa, 1 Tomoya Iwakura, 1 Kimi Kaneko, 2∗Hiroaki Yoshida, 1 Yasutaka Kumano, 3 Kazutaka Shimada, 4 Rafal Rzepka, 5 Patrycja Swieczkowska 5 1 Fujitsu Limited, Tokyo, Japan 2 Toyota Motor Corporation, Aichi, Japan 3 G-Search Limited, Tokyo, Japan 4 Kyushu Institute of Technology, Fukuoka, Japan 5 Hokkaido University, Hokkaido, Japan 1,3 {y.hiyori,iwakura.tomoya,yoshida.hiro-15,kumano.yasutaka}@fujitsu.com 2 kimi kaneko@mail.toyota.co.jp 4 shimada@ai.kyutech.ac.jp 5 {rzepka,swieczkowska}@ist.hokudai.ac.jp Abstract This paper investigates the effectiveness of au- tomatic annotator assignment for text annota- tion in expert domains. In the task of creat- ing high-quality annotated corpora, expert do- mains often cover multiple sub-domains (e.g. organic and inorganic chemistry in the chem- istry domain) either explicitly or implicitly. Therefore, it is crucial to assign annotators to documents relevant with their fine-grained domain expertise. However, most of exist- ing methods for crowdsoucing estimate relia- bility of each annotator or annotated instance only after the annotation process. To address the issue, we propose a method to estimate the domain expertise of each annotator before the annotation process using information eas- ily available from the annotators beforehand. We propose two measures to estimate the anno- tator expertise: an explicit measure using the predefined categories of sub-domains, and an implicit measure using distributed representa- tions of the documents. The experimental re- sults on chemical name annotation tasks show that the annotation accuracy improves when both explicit and implicit measures for anno- tator assignment are combined. 1 Introduction Preparation of training data has been a critical is- sue in applying the supervised and semi-supervised methods to real world problems. Training data con- struction is often quite costly and the quality is hard to assure, especially when the task requires expert knowledge of the target domain such as chemistry and medicine. A possible solution to alleviate the lack of annotated data is the use of a crowdsourc- ing platform. Crowdsourced annotation has proven to be successful for lowering the annotation costs in tasks that require general knowledge, such as ∗Work done while the author was at Fujitsu Laboratories Ltd. POS tagging (Hovy et al., 2014), textual entailment, word sense disambiguation (Snow et al., 2008), or recognition of general named entities like PERSON and ORGANIZATION (Finin et al., 2010; Nguyen et al., 2017). Meanwhile, there is limited success in using crowdsourcing for annotation in expert domains mainly because of the poor annotation quality made by annotators with low domain exper- tise. In the medical domain, for example, Nye et al. (2018) combine both non-expert crowdsourced an- notators and expert annotators for text annotation. We hypothesize that a key challenge of crowd- sourced annotation in expert domains lies in the difficulty of estimating the candidate annotators’ expertise in terms of the relevance to the documents to be annotated. Some crowdsourcing platforms such as Amazon Mechanical Turk 1 provide the feature to specify workers that have the expertise in …

Content truncated to 500 words