Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5677–5697 July 9-14, 2023 ©2023 Association for Computational Linguistics Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification Sha Li1, Ruining Zhao1, Manling Li1, Heng Ji1, Chris Callison-Burch2, Jiawei Han1 1University of Illinois at Urbana-Champaign 2 University of Pennsylvania {shal2, ruining9, manling2, hengji, hanj}@illinois.edu ccb@seas.upenn.edu Abstract Event schemas are a form of world knowledge about the typical progression of events. Re- cent methods for event schema induction use information extraction systems to construct a large number of event graph instances from documents, and then learn to generalize the schema from such instances. In contrast, we propose to treat event schemas as a form of commonsense knowledge that can be derived from large language models (LLMs). This new paradigm greatly simplifies the schema induc- tion process and allows us to handle both hierar- chical relations and temporal relations between events in a straightforward way. Since event schemas have complex graph structures, we de- sign an incremental prompting and verification method INCSCHEMA to break down the con- struction of a complex event graph into three stages: event skeleton construction, event ex- pansion, and event-event relation verification. Compared to directly using LLMs to generate a linearized graph, INCSCHEMA can generate large and complex schemas with 7.2% F1 im- provement in temporal relations and 31.0% F1 improvement in hierarchical relations. In addi- tion, compared to the previous state-of-the-art closed-domain schema induction model, hu- man assessors were able to cover ∼10% more events when translating the schemas into co- herent stories and rated our schemas 1.3 points higher (on a 5-point scale) in terms of readabil- ity. 1 1 Introduction Schemas, defined by (Schank and Abelson, 1975) as “a predetermined, stereotyped sequence of ac- tions that defines a well-known situation”, are a manifestation of world knowledge. With the help of schemas, a model can then infer missing events such as a person must have “been within contact with a pathogen” before the event “the person was 1Code and ODIN dataset available at https:// github.com/raspberryice/inc-schema. Instance Graphs Entity Extraction Event Extraction Coreference Relation Extraction IE System Documents Schema Graph Large Language Model Incremental prompting & veri�cation Graph Learning Model Figure 1: A comparison between the instance-based schema induction pipeline (gray background) and our INCSCHEMA approach. By directly prompting LLMs to construct the schema graph, our framework is con- ceptually simpler, open-domain, extensible, and more interpretable. sent to the hospital for treatment” and also pre- dict that if a large-scale incident happened, this might trigger an “investigation of the source of the pathogen”. To automate schema creation, two mainstream approaches are to learn from manually created ref- erence schemas or learn from large amounts of event instances automatically extracted from doc- uments. Manual creation of complex hierarchi- cal schemas requires expert annotation, which is not scalable2. On the other hand, instance-based schema induction methods (Li et al., 2020, 2021; Jin et al., 2022; Dror et al., 2022) rely on …

Content truncated to 500 words