Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2761–2772 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2761 Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domains Claudia Schulz1, Christian M. Meyer1, Jan Kiesewetter2, Michael Sailer3, Elisabeth Bauer3, Martin R. Fischer2, Frank Fischer3, and Iryna Gurevych1 1 Ubiquitous Knowledge Processing (UKP) Lab, Technische Universit¨at Darmstadt, Germany 2 Institute of Medical Education, University Hospital, LMU M¨unchen, Germany 3 Chair of Education and Educational Psychology, LMU M¨unchen, Germany http://famulus-project.de Abstract Many complex discourse-level tasks can aid domain experts in their work but require costly expert annotations for data creation. To speed up and ease annotations, we investigate the vi- ability of automatically generated annotation suggestions for such tasks. As an example, we choose a task that is particularly hard for both humans and machines: the segmenta- tion and classification of epistemic activities in diagnostic reasoning texts. We create and publish a new dataset covering two domains and carefully analyse the suggested annota- tions. We find that suggestions have positive effects on annotation speed and performance, while not introducing noteworthy biases. En- visioning suggestion models that improve with newly annotated texts, we contrast methods for continuous model adjustment and suggest the most effective setup for suggestions in future expert tasks. 1 Introduction Current deep learning methods require large amounts of training data to achieve reasonable performance. Scalable solutions to acquire la- belled data use crowdsourcing (e.g., Potthast et al., 2018), gamification (Ahn, 2006), or incidental su- pervision (Roth, 2017). For many complex tasks in expert domains, such as law or medicine, this is, however, not an option since crowdworkers and gamers lack the necessary expertise. Annotating data manually is therefore often the only way to train a model for tasks aiding experts with their work. But the more expertise an annotation task requires, the more time- and funding-intensive it typically is, which is why many projects suffer from small corpora and deficient models. In this paper, we propose and analyse an annota- tion setup aiming to increase the annotation speed and ease for a discourse-level sequence labelling task requiring extensive domain expertise, without sacrificing annotation quality. For the first time, we study the effects of automatically suggesting annotations to expert annotators in a task that is hard for both humans (only moderate agreement) and machine learning models (only mediocre per- formance) and compare the effects across differ- ent domains and suggestion models. We further- more investigate how the performance of the mod- els changes if they continuously learn from expert annotations. As our use case, we consider the task of an- notating epistemic activities in diagnostic rea- soning texts, which was recently introduced by Schulz et al. (2018, 2019). The task is theoret- ically grounded in the learning sciences (Fischer et al., 2014) and enables innovative applications that teach diagnostic skills to university students based on automatically generated feedback about their reasoning processes. This task is an ideal choice for …

Content truncated to 500 words