Automatic Error Analysis for Document-level Information Extraction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3960 - 3975 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Automatic Error Analysis for Document-level Information Extraction Aliva Das∗, Xinya Du∗, Barry Wang∗, Kejian Shi, Jiayuan Gu, Thomas Porter, Claire Cardie Department of Computer Science, Cornell University {ad677, xd75, zw545, ks2325, jg844, tjp78, ctc9}@cornell.edu Abstract Document-level information extraction (IE) tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automat- ing error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-the- art document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation.1 1 Introduction Although information extraction (IE) research has almost uniformly focused on sentence-level rela- tion and event extraction (Grishman, 2019), the earliest research in the area formulated the task at the document level. Consider, for example, the first large-scale (for the time) evaluations of IE systems — e.g. MUC-3 (1991) and MUC-4 (1992). Each involved a complex document-level event extrac- tion task: there were 24 types of events, over a dozen event arguments (or roles) to be identified for each event; documents could contain zero to tens of events, and extracting argument entities (or role fillers) required noun phrase coreference res- olution to ensure interpretability for the end-user (e.g. to ensure that multiple distinct mentions of the *These authors contributed equally to this work. 1Our code for the error analysis tool and its output on dif- ferent model predictions are available at https://github. com/IceJinx33/auto-err-template-fill/. same entity in the output were not misinterpreted as references to distinct entities). The task was challenging: information relevant for a single event could be scattered across the document or repeated in multiple places; relevant information might need to be shared across multi- ple events; information regarding different events could be intermingled. In Figure 1, for example, the DISEASE "Newcastle" is mentioned well before its associated event is mentioned (via the triggering phrase "the disease has killed"); that same mention of "Newcastle" must again be recognized as the DISEASE in a second event; and the COUNTRY of the first event ("Honduras") appears only in the sentence describing the second event. In fact, the problem of document-level informa- tion extraction has only recently begun to be re- visited (Quirk and Poon, 2017; Jain et al., 2020; Du et al., 2021b,a; Li et al., 2021; Du, 2021; Yang et al., 2021) in part in an attempt to test the power of end-to-end neural network techniques that have been so successful on …

Content truncated to 500 words