ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2559 - 2575 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification Yucheng Zhou1∗, Tao Shen2, Xiubo Geng2†, Guodong Long1, Daxin Jiang2† 1Australian AI Institute, School of CS, FEIT, University of Technology Sydney 2Microsoft yucheng.zhou.uts@gmail.com, guodong.long@uts.edu.au {shentao, xigeng, djiang}@microsoft.com Abstract Generating new events given context with cor- related ones plays a crucial role in many event- centric reasoning tasks. Existing works either limit their scope to specific scenarios or over- look event-level correlations. In this paper, we propose to pre-train a general Correlation- aware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event- correlation encoding and prompt-based event locating, which highlight event-level correla- tions with effective training. The proposed ClarET is applicable to a wide range of event- centric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application for- mulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, coun- terfactual and ending reasoning). Empirical fine-tuning results, as well as zero- and few- shot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability. 1 Introduction An ‘event’, usually a text span composed of a pred- icate and its arguments (Zhang et al., 2020b), is a fine-grained semantic unit to describe the state of entities/things (e.g., He looks very worried) and how they act (e.g., I grab his arms). Understanding events and modeling their correlations are funda- mental to many reasoning tasks (Bhagavatula et al., 2020; Qin et al., 2019), e.g., abductive reasoning, story ending classification and generation, counter- factual reasoning, script reasoning. For instance, in the left example of Figure 1, to generate the miss- ing event [E] in the given context, it is essential to understand that there are four events (‘it tries ∗Work is done during internship at Microsoft. †Corresponding author. Context: It tries the knob but [E], so the creature starts pounding on the door to break it down. Paragraph 𝑥: It tries the knob but it’s locked, so the creature starts pounding on … An event mention 𝑒: it’s locked Negative events { ҧ𝑒}𝑖=1 𝑀 : it’s smoked he’s gone … Output: it’s locked. Figure 1: Left: an example of abductive reasoning which aims to generate the missing event [E] given correlated events (underlined) and connectives (w/ orange) in the context. Right: a toy example (x, e, {¯e}M i=1) of event-rich data for better read- ing. See Appendix A for real examples to pre-train our model. the knob’, [E], ‘the creature starts pounding on the door’, and ‘(the creature) to break it down’), and then predict [E] based on the other three events and its correlations to them (i.e., the contrast relation indicated by ‘but’ and the causal relation by ‘so’). Event-aware reasoning has gained …

Content truncated to 500 words