DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3970 - 3982 July 10-15, 2022 ©2022 Association for Computational Linguistics DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction Meihan Tong1, Bin Xu1∗, Shuai Wang2, Meihuan Han1, Yixin Cao3 Jiangqi Zhu1, Siyu Chen1, Lei Hou1, Juanzi Li1 1Knowledge Engineering Laboratory, Tsinghua University, Beijing, China 2SLP Group, AI Technology Department, JOYY Inc, China 3Singapore Management University, Singapore tongmeihan@gmail.com, xubin@tsinghua.edu.cn Abstract Event extraction aims to identify an event and then extract the arguments participating in the event. Despite the great success in sentence- level event extraction, events are more natu- rally presented in the form of documents, with event arguments scattered in multiple sentences. However, a major barrier to promote document- level event extraction has been the lack of large-scale and practical training and evalua- tion datasets. In this paper, we present Do- cEE, a new document-level event extraction dataset including 27,000+ events, 180,000+ ar- guments. We highlight three features: large- scale manual annotations, fine-grained argu- ment types and application-oriented settings. Experiments show that there is still a big gap between state-of-the-art models and human beings (41% Vs 85% in F1 score), indicat- ing that DocEE is an open issue. DocEE is now available at https://github.com/ tongmeihan1995/DocEE.git. 1 Introduction Event Extraction (EE) aims to detect events from text, including event classification and event argu- ment extraction. EE is one of the fundamental tasks in text mining (Feldman and Sanger, 2006) and has many applications. For instance, it can monitor political or military crises to generate real-time notifications and alerts (Dragos, 2013), and dig the links and connections (e.g., Who Met Whom and When) between dignitaries for portrait analysis (Zhan et al., 2020). Most existing datasets (e.g., ACE2005 1 and KBP2017 2) focus on sentence-level event extrac- tion, while events are usually described at the docu- ment level, and event arguments are typically scat- tered across different sentences (Hamborg et al., ∗Corresponding author. 1https://catalog.ldc.upenn.edu/LDC2006T06 2https://tac.nist.gov/2017/KBP/ 2019). Figure 1 shows an Air Crash event. To ex- tract argument Date, we need to read sentence [1], while to extract argument Cause of the Accident, we need to integrate information in sentences [6] and [7]. Clearly, this requires reasoning over multiple sentences and modeling long-distance dependency, intuitively beyond the reach of sentence-level EE. Therefore, it is necessary to move EE forward from sentence-level to document-level. Only a few datasets are curated for document- level EE. MUC-4(Grishman and Sundheim, 1996) provides 1,700 news articles annotated with 4 event types and 5 argument types. The 5 arguments are shared among different event types without further refinement. WikiEvents(Li et al., 2021) consists of only 246 documents with very few (22% of total) cross-sentences argument annota- tions. RAMS(Ebner et al., 2020) limits the scope of the arguments in a 5-sentence window around its event trigger, which is not in line with the ac- tual application, and the number of the argument types in RAMS is only 65, which …

Content truncated to 500 words