Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5188–5197 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5188 Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation Yuval Marton1, Asad Sayeed2 1University of Washington, WA, USA 2Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg, Sweden ymarton@uw.edu, asad.sayeed@gu.se Abstract Modeling thematic fit (a verb–argument compositional semantics task) currently requires a very large burden of labeled data. We take a linguistically machine-annotated large corpus and replace corpus layers with output from higher-quality, more modern taggers. We compare the old and new corpus versions’ impact on a verb–argument fit modeling task, using a high-performing neural approach. We discover that higher annotation quality dramatically reduces our data requirement while demonstrating better supervised predicate-argument classification. But in applying the model to psycholinguistic tasks outside the training objective, we see clear gains at scale, but only in one of two thematic fit estimation tasks, and no clear gains on the other. We also see that quality improves with training size, but perhaps plateauing or even declining in one task. Last, we tested the effect of role set size. All this suggests that the quality/quantity interplay is not all you need. We replicate previous studies while modifying certain role representation details and set a new state-of-the-art in event modeling, using a fraction of the data. We make the new corpus version public. Keywords: SRL, thematic fit, psycholinguistics 1. Introduction Is more data always more effective than better anno- tation? Is it always cheaper just to obtain and use data with mid-quality annotation than improve anno- tation quality over a smaller dataset? Traditionally, to researchers grounded in linguistics, it seemed obvious that higher quality and richer annotation should be bet- ter. But with the advent of “Big Data”, the common wisdom seem to have shifted toward more data; Deep Learning continued this trend (see examples in Sec- tion 1.1). We re-examine these questions using two types of nat- ural language processing (NLP) tasks: (1) supervised thematic role prediction (given a predicate and an ar- gument’s word span; here only its syntactic head word) and word prediction (given a predicate and a role); and (2) psycholinguistic tasks outside the explicit training objective: rating the thematic fit between a verb and its potential arguments. These tasks have a large body of work in computational linguistics (see section 1.2). We examine the trade-offs in training models designed to accomplish these tasks through modeling events and their participants in a large corpus (task (1) above). The trade-off we focus on is using more data with mediocre linguistic annotations versus little data with higher-quality annotations. For the former, we replicate a PropBank-based model of Hong et al. (2018) using increasing subsets of their training data, a large corpus with machine-predicted annotations of mediocre qual- ity. For the latter, we replace some annotation lay- ers with equivalent layers generated by higher qual- ity linguistic tools. Our model implementation …

Content truncated to 500 words