Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3064–3074 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3064 Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports Markus Zlabinger TU Wien, Vienna, Austria markus.zlabinger@tuwien.ac.at Marta Sabou TU Wien, Vienna, Austria marta.sabou@tuwien.ac.at Sebastian Hofst¨atter TU Wien, Vienna, Austria s.hofstaetter@tuwien.ac.at Allan Hanbury TU Wien, Vienna, Austria allan.hanbury@tuwien.ac.at Abstract The search for Participants, Interventions, and Outcomes (PIO) in clinical trial reports is a critical task in Evidence Based Medicine. For an automatic PIO extraction, high-quality cor- pora are needed. Obtaining such a corpus from crowdworkers, however, has been shown to be ineffective since (i) workers usually lack domain-specific expertise to conduct the task with sufficient quality, and (ii) the standard ap- proach of annotating entire abstracts of trial reports as one task-instance (i.e. HIT) leads to an uneven distribution in task effort. In this paper, we switch from entire abstract to sentence annotation, referred to as the SEN- BASE approach. We build upon SENBASE in SENSUPPORT, where we compensate the lack of domain-specific expertise of crowdwork- ers by showing for each task-instance simi- lar sentences that are already annotated by ex- perts. Such tailored task-instance examples are retrieved via unsupervised semantic short- text similarity (SSTS) method – and we evalu- ate nine methods to find an effective solution for SENSUPPORT. We compute the Cohen’s Kappa agreement between crowd-annotations and gold standard annotations and show that (i) both sentence-based approaches outperform a BASELINE approach where entire abstracts are annotated; (ii) supporting annotators with tai- lored task-instance examples is the best per- forming approach with Kappa agreements of 0.78/0.75/0.69 for P, I, and O respectively. 1 Introduction Evidence Based Medicine is the practice of deci- sion making based on the best available scientific information. Finding such information rapidly is essential, especially in the current pandemic crisis where thousands of medical articles about COVID- 19 are published weekly (ˇSkori´c et al., 2020). To make the search process time-efficient, the PICO model enables specific search for: Participants (e.g. “patients with headache”), Interventions (“ibupro- fen”), Comparisons (“placebo”), and Outcomes (“pain reduction”) (Huang et al., 2006). To allow a search for structured PICO information in trial reports, a prior automatic extraction is necessary. The effectiveness of an automatic PICO extrac- tion depends on the quality of manually annotated corpora. As an alternative to scarce and expen- sive expert annotators, Nye et al. (2018) hired crowdworkers from the Mechanical Turk platform (MTurk) to annotate Participants, Interventions, and Outcomes (PIO1) in clinical trial reports. The crowdworkers, however, reached low agreements to expert annotations, potentially affected by (i) a lack of domain-specific expertise of the crowd- workers, and (ii) an uneven task length distribution. The lack of domain-specific expertise makes it difficult for crowdworkers to understand the termi- nology and jargon that prevails in medical litera- ture (Kim et al., 2011; Wallace, 2018). As a result, workers experience medical tasks as cognitively overwhelming, with the side effect of a decreased label quality (Finnerty …

Content truncated to 500 words