CLEVE: Contrastive Pre-training for Event Extraction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6283–6297 August 1–6, 2021. ©2021 Association for Computational Linguistics 6283 CLEVE: Contrastive Pre-training for Event Extraction Ziqi Wang1∗, Xiaozhi Wang1∗, Xu Han1, Yankai Lin3, Lei Hou1,2† , Zhiyuan Liu1,2, Peng Li3, Juanzi Li1,2, Jie Zhou3 1Department of Computer Science and Technology, BNRist; 2KIRC, Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China 3Pattern Recognition Center, WeChat AI, Tencent Inc, China {ziqi-wan16, wangxz20, hanxu17}@mails.tsinghua.edu.cn Abstract Event extraction (EE) has considerably ben- efited from pre-trained language models (PLMs) by fine-tuning. However, existing pre-training methods have not involved mod- eling event characteristics, resulting in the de- veloped EE models cannot take full advan- tage of large-scale unsupervised data. To this end, we propose CLEVE, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) ob- tained with automatic parsers. CLEVE con- tains a text encoder to learn event seman- tics and a graph encoder to learn event struc- tures respectively. Specifically, the text en- coder learns event semantic representations by self-supervised contrastive learning to rep- resent the words of the same events closer than those unrelated words; the graph en- coder learns event structure representations by graph contrastive pre-training on parsed event- related semantic structures. The two com- plementary representations then work together to improve both the conventional supervised EE and the unsupervised “liberal” EE, which requires jointly extracting events and discov- ering event schemata without any annotated data. Experiments on ACE 2005 and MAVEN datasets show that CLEVE achieves significant improvements, especially in the challenging unsupervised setting. The source code and pre-trained checkpoints can be obtained from https://github.com/THU-KEG/CLEVE. 1 Introduction Event extraction (EE) is a long-standing crucial in- formation extraction task, which aims at extracting event structures from unstructured text. As illus- trated in Figure 1, it contains event detection task to identify event triggers (the word “attack”) and ∗ indicates equal contribution † Correspondence to L.Hou (houlei@tsinghua.edu.cn) CNN's Kelly Wallace reports on today's attack in Netanya. Attack Time- within Place Event Type Argument Role Attacker Target Text CNN’s Kelly Wallace report-01 attack-01 today Netanya ARG0 ARG1 ARG1 time AMR Structure Event Schema Figure 1: An example sampled from the ACE 2005 dataset with its event annotation and AMR structure. classify event types (Attack), as well as event argument extraction task to identify entities serv- ing as event arguments (“today” and “Netanya”) and classify their argument roles (Time-within and Place) (Ahn, 2006). By explicitly captur- ing the event structure in the text, EE can benefit various downstream tasks such as information re- trieval (Glavaˇs and ˇSnajder, 2014) and knowledge base population (Ji and Grishman, 2011). Existing EE methods mainly follow the supervised-learning paradigm to train advanced neural networks (Chen et al., 2015; Nguyen et al., 2016; Nguyen and Grishman, 2018) with human- annotated datasets and pre-defined event schemata. These methods work well in lots of public bench- marks …

Content truncated to 500 words