Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2148–2156 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2148 Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training Merel C.J. Scholmana, Valentina Pyatkinb, Frances Yunga, Ido Daganb, Reut Tsarfatyb, Vera Demberga aSaarland University, bBar Ilan University Saarbr¨ucken, Germany; Ramat Gan, Israel {m.c.j.scholman,frances,vera}@coli.uni-saarland.de {pyatkiv,reut.tsarfaty}@biu.ac.il; dagan@cs.biu.ac.il Abstract Obtaining linguistic annotation from novice crowdworkers is far from trivial. A case in point is the annotation of discourse relations, which is a complicated task. Recent methods have obtained promising results by extracting relation labels from either discourse connectives (DCs) or question-answer (QA) pairs that participants provide. The current contribution studies the effect of worker selection and training on the agreement on implicit relation labels between workers and gold labels, for both the DC and the QA method. In Study 1, workers were not specifically selected or trained, and the results show that there is much room for improvement. Study 2 shows that a combination of selection and training does lead to improved results, but the method is cost- and time-intensive. Study 3 shows that a selection-only approach is a viable alternative; it results in annotations of comparable quality compared to annotations from trained participants. The results generalized over both the DC and QA method and therefore indicate that a selection-only approach could also be effective for other crowdsourced discourse annotation tasks. Keywords: discourse annotations, crowdsourcing, training, participant selection 1. Introduction Obtaining linguistic annotations from novice crowd- workers is far from trivial. A case in point is the an- notation of discourse relations, which is a complicated task. Discourse relations (DRs) are logical relations that readers infer between segments of a text, such as Cause or Contrast. Classifying these relations into cat- egories relies on annotators’ interpretation of a text, which makes it a particularly difficult task, as reflected in relatively low inter-annotator agreement (Spooren and Degand, 2010). An additional complicating factor is the variety of rela- tion types that can be distinguished: The linguistic for- malism used in the current study, Penn Discourse Tree- bank 3.0 (PDTB) (Webber et al., 2019), distinguishes 36 unique types. A thorough understanding of these distinctions requires expert knowledge. DR annota- tions are therefore traditionally obtained from expert, trained annotators. However, employing such annota- tors is time-consuming and costly, and so researchers have looked at other ways to obtain reliable annota- tions, such as crowdsourcing. Crowdsourced studies typically consist of easier and more intuitive tasks, which can be performed by lay- men. Crowdsourcing allows for fast and cost-effective collection of labelled data, but because the tasks need to be intuitive, crowdworkers cannot be asked to code according to a specific linguistic formalism such as PDTB. Further, with such complex tasks, one would need to make sure that the crowdworkers can reach an acceptable level of proficiency. Annotation tasks are often made suitable for crowd- sourcing by considering two dimensions: worker se- lection and training procedures, …

Content truncated to 500 words