Leveraging Active Learning to Minimise SRL Annotation Across Corpora

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 399–408 July 13-14, 2023 ©2023 Association for Computational Linguistics Leveraging Active Learning to Minimise SRL Annotation Across Corpora Skatje Myers University of Colorado at Boulder skatje.myers@colorado.edu Martha Palmer University of Colorado at Boulder mpalmer@colorado.edu Abstract In this paper we investigate the application of active learning to semantic role labeling (SRL) using Bayesian Active Learning by Dis- agreement (BALD). Our new predicate-focused selection method quickly improves efficiency on three different specialised domain corpora. This is encouraging news for researchers want- ing to port SRL to domain specific applica- tions. Interestingly, with the large and diverse OntoNotes corpus, the sentence selection ap- proach, that collects a larger number of predi- cates, taking more time to annotate, fares better than the predicate approach. In this paper, we analyze both the selections made by our two selections methods for the various domains and the differences between these corpora in detail. 1 Introduction The majority of natural language processing (NLP) systems are reliant on manual annotations to train supervised models. Although semi-supervised and unsupervised methods are frequently employed to help adapt models to new domains, human anno- tation remains the gold standard for quality input. Due to the high cost of human annotation, espe- cially if the task requires expert knowledge, and the time-intensive process, this can be daunting for many applications. Active learning (AL) has been shown to reduce annotation requirements for a variety of NLP tasks (Zhang et al., 2022) by selecting more informative instances that are most likely to fill gaps in the model’s knowledge. In this paper, we focus specifically on the NLP task of semantic role labeling (SRL). The goal of SRL is to identify and label the who, what, and when of predicates in a sentence. This information can be used as features in downstream applications such as information extraction (MacAvaney et al., 2017), machine translation (Marcheggiani et al., 2018), and features prominently in Abstract Mean- ing Representation (AMR) applications (Banarescu et al., 2013). In this paper, we propose a new selection strategy tuned for SRL that is based off of previous methods of using model dropout to approximate a Gaussian process (Siddhant and Lipton, 2018). We compare this to prior work on AL selection for SRL (Myers and Palmer, 2021) on four corpora in a variety of domains: ecology, earthquakes, clinical notes, and the large multi-genre OntoNotes corpus. Since sentences in most domains typically con- tain multiple predicates, there are often redundan- cies in choosing predicates to annotate on the sen- tence level. Although a sentence may contain a particularly informative predicate, annotating high- frequency verbs such as "be" that co-occur in the sentence may not be beneficial. We instead use a method to select specific predicate-argument struc- tures and compare the impact on performance as compared to selecting whole sentences instead. This method is a natural extension that allows us to even better leverage the focused annotation that active learning offers by …

Content truncated to 500 words