Revisiting the Evaluation of End-to-end Event Extraction

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4609–4617 August 1–6, 2021. ©2021 Association for Computational Linguistics 4609 Revisiting the Evaluation of End-to-end Event Extraction Shun Zheng†, Wei Cao†, Wei Xu‡, Jiang Bian† †Microsoft Research ‡Institute of Interdisciplinary Information Sciences, Tsinghua University {shun.zheng, wei.cao, jiang.bian}@microsoft.com; weixu@mail.tsinghua.edu.cn Abstract Event extraction (EE) aims to harvest event in- stances from plain text, where each instance is composed of a group of event arguments with specific event roles. Existing end-to-end EE re- search usually adopts the role-averaged evalu- ation that produces evaluation measures by av- eraging evaluation statistics of each event role. However, although this averaged metric can in- dicate the model performance to some extent, we find that such metric can be pretty mis- leading to downstream applications that uti- lize an event instance as a whole, where one wrongly identified event argument can substan- tially alter the whole meaning of an event in- stance. To mitigate this gap and provide a more complete understanding of performance, we propose two new evaluation metrics that also consider an event instance as a whole and explicitly penalize wrongly identified event ar- guments. Moreover, to support diverse pref- erences of evaluation metrics motivated by different scenarios, we propose a new train- ing paradigm based on reinforcement learn- ing for a typical end-to-end EE model, i.e., Doc2EDAG. Our extensive experiments show that the new training improves the initial one by a large margin (about 10%) under new met- rics. Nevertheless, the current performance is still far from satisfactory, and optimizing to- wards these new metrics calls for more future research. 1 Introduction Event extraction (EE) is a vital task that aims to har- vest structured event instances from unstructured plain text. Such structured knowledge can bene- fit many downstream applications, such as ques- tion answering, language understanding, knowl- edge graph, etc. In general, an event instance is composed of a group of entities (person, organiza- tion, date, etc.) that jointly describes an incident. Each entity of the event instance, also referred to Event Schema The Gap E3 E2 E1 Event Table E2 E4 E1 Role 3 Role 2 Role 1 Ground Truth E2 E1 Role 3 Role 2 Role 1 Prediction E3 E1 E2 E3 Text David held 36,000 shares of the company. Initially, a depository froze all his shares until Sept. 9th, 2019. But the unfrozen date was extended to Dec. 9th, 2019 due to … E1: David E2: 36,000 E3: Sept. 9th, 2019 E4: Dec. 9th, 2019 Entity Info Equity Freeze (Event Type) Stockholder (Role 1) Frozen Shares (Role 2) Unfrozen Date (Role 3) Role-averaged Evaluation Practical Utilization Figure 1: An example to illustrate the gap between the role-averaged evaluation and a practical application that utilizes an event instance as a whole, where “E1”, “E2”, “E3”, and “E4” are entity marks, “Role 1”, “Role 2”, and “Role 3” represent event roles, and “(E1, E2, E4)” is an event instance. We can see that the wrong ar- gument “E3” makes the whole event instance …

Content truncated to 500 words