Word-Label Alignment for Event Detection: A New Perspective via Optimal Transport

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 1 - 11 July 14-15, 2022 ©2022 Association for Computational Linguistics What do Large Language Models Learn about Scripts? Abhilasha Sancheti University of Maryland, College Park Adobe Research sancheti@{umd.edu,adobe.com} Rachel Rudinger University of Maryland, College Park rudinger@umd.edu Abstract Script Knowledge (Schank and Abelson, 1975) has long been recognized as crucial for lan- guage understanding as it can help in filling in unstated information in a narrative. However, such knowledge is expensive to produce man- ually and difficult to induce from text due to reporting bias (Gordon and Van Durme, 2013). In this work, we are interested in the scientific question of whether explicit script knowledge is present and accessible through pre-trained generative language models (LMs). To this end, we introduce the task of generating full event sequence descriptions (ESDs) given a sce- nario as a natural language prompt. Through zero-shot probing, we find that generative LMs produce poor ESDs with mostly omitted, irrele- vant, repeated or misordered events. To address this, we propose a pipeline-based script induc- tion framework (SIF) which can generate good quality ESDs for unseen scenarios (e.g., bake a cake). SIF is a two-staged framework that fine-tunes LM on a small set of ESD examples in the first stage. In the second stage, ESD gen- erated for an unseen scenario is post-processed using RoBERTa-based models to filter irrele- vant events, remove repetitions, and reorder the temporally misordered events. Through auto- matic and manual evaluations, we demonstrate that SIF yields substantial improvements (1-3 BLEU points) over a fine-tuned LM. However, manual analysis shows that there is great room for improvement, offering a new research di- rection for inducing script knowledge1. 1 Introduction Scripts are structured commonsense knowledge in the form of event sequences that characterize com- monplace scenarios, such as, eating at a restau- rant (Schank and Abelson, 1975). Scripts are fun- damental pieces of commonsense knowledge that humans share and assume to be tacitly understood 1Code and dataset are available at https://github. com/abhilashasancheti/script-generation Figure 1: Sample event sequence description (ESD) from Wanzare et al. (2016) for BAKING A CAKE sce- nario. We use natural language prompts (Table 2) to generate completely ordered ESDs for evaluating extent of script knowledge accessible through LMs. by each other. When someone says “I went to a restaurant for lunch", our script knowledge allows us to infer that a waiter would have taken the order, the speaker would have eaten the lunch, payed for it, and tipped the waiter, even if these events are not explicitly mentioned. Knowledge of scripts, whether implicit or explicit, has been recognized as important for language understanding tasks (Mi- ikkulainen, 1995; Mueller, 2004). Earlier efforts to automatically induce scripts from text on a large scale include Chambers and Jurafsky (2008) who treat the problem of script in- duction as one of learning narrative chains using textual co-occurrence statistics. However, report- ing bias (Gordon and Van Durme, 2013) remains an obstacle for script induction as many …

Content truncated to 500 words