GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3664–3686 July 9-14, 2023 ©2023 Association for Computational Linguistics GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles Tanmay Parekh† I-Hung Hsu‡ Kuan-Hao Huang† Kai-Wei Chang† Nanyun Peng† †Computer Science Department, University of California, Los Angeles ‡Information Science Institute, University of Southern California {tparekh, khhuang, kwchang, violetpeng}@cs.ucla.edu {ihunghsu}@isi.edu Abstract Recent works in Event Argument Extraction (EAE) have focused on improving model generalizability to cater to new events and domains. However, standard benchmarking datasets like ACE and ERE cover less than 40 event types and 25 entity-centric argument roles. Limited diversity and coverage hinder these datasets from adequately evaluating the generalizability of EAE models. In this paper, we first contribute by creating a large and diverse EAE ontology. This ontology is created by transforming FrameNet, a comprehensive semantic role labeling (SRL) dataset for EAE, by exploiting the similarity between these two tasks. Then, exhaustive human expert annotations are collected to build the ontology, concluding with 115 events and 220 argument roles, with a significant portion of roles not be- ing entities. We utilize this ontology to further introduce GENEVA, a diverse generalizability benchmarking dataset comprising four test suites, aimed at evaluating models’ ability to handle limited data and unseen event type generalization. We benchmark six EAE models from various families. The results show that owing to non-entity argument roles, even the best-performing model can only achieve 39% F1 score, indicating how GENEVA provides new challenges for generalization in EAE. Overall, our large and diverse EAE ontology can aid in creating more comprehensive future resources, while GENEVA is a challenging benchmarking dataset encouraging further research for improving generalizability in EAE. The code and data can be found at https: //github.com/PlusLabNLP/GENEVA. 1 Introduction Event Argument Extraction (EAE) aims at extract- ing structured information of event-specific argu- ments and their roles for events from a pre-defined taxonomy. EAE is a classic topic (Sundheim, 1992) and elemental for a wide range of applications like building knowledge graphs (Zhang et al., 2020), Figure 1: Distribution of event types into various ab- stract event types1 for GENEVA, ACE, ERE, RAMS, and WikiEvents datasets. We observe that GENEVA is relatively more diverse than the other datasets. question answering (Berant et al., 2014), and others (Hogenboom et al., 2016; Wen et al., 2021; Yang et al., 2019b). Recent works have focused on build- ing generalizable EAE models (Huang et al., 2018; Lyu et al., 2021; Sainz et al., 2022) and they uti- lize existing datasets like ACE (Doddington et al., 2004) and ERE (Song et al., 2015) for benchmark- ing. However, as shown in Figure 1, these datasets have limited diversity as they focus only on two abstract types,1 Action and Change. Furthermore, they have restricted coverage as they only comprise argument roles that are entities. The limited diver- sity and coverage restrict the ability of these exist- ing datasets to robustly evaluate the generalizability of …

Content truncated to 500 words