Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction!

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3689–3701, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3689 Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction! Bruno Taill´e1,2, Vincent Guigue1, Geoffrey Scoutheeten2 and Patrick Gallinari1,3 1Sorbonne Universit´e, CNRS, Laboratoire d’Informatique de Paris 6, LIP6 2BNP Paribas 3Criteo AI Lab {bruno.taille, vincent.guigue, patrick.gallinari}@lip6.fr geoffrey.scoutheeten@bnpparibas.com Abstract Despite efforts to distinguish three different evaluation setups (Bekoulis et al., 2018a,b), numerous end-to-end Relation Extraction (RE) articles present unreliable performance com- parison to previous work. In this paper, we first identify several patterns of invalid compar- isons in published papers and describe them to avoid their propagation. We then propose a small empirical study to quantify the most common mistake’s impact and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. We also seize this opportunity to study the unexplored ablations of two recent developments: the use of lan- guage model pretraining (specifically BERT) and span-level NER. This meta-analysis em- phasizes the need for rigor in the report of both the evaluation setting and the dataset statistics. We finally call for unifying the evaluation set- ting in end-to-end RE 1. 1 Introduction Named Entity Recognition (NER)2 and Relation Extraction (RE) are key Information Extraction tasks, for example at the heart of Knowledge Graph Construction along with Coreference Resolution and Entity Linking. In the traditional pipeline ap- proach, these tasks are treated with two models trained separately and applied sequentially (Bach and Badaskar, 2007). Nevertheless, combining information from both submodules is beneficial (Roth and Yih, 2002) and end-to-end RE models tackling both tasks jointly have been proposed to better model their interdependency and overcome cascading errors (Li and Ji, 2014). This end-to-end setting has recently received more attention in the wake of improved language models (LM). However, in this prolific and compet- itive domain, authors have used several evaluation settings to compare their performance. And despite the attempt to clearly identify three main setups (Bekoulis et al., 2018a,b), this multiplication of settings makes the apprehension of the literature difficult and confusing, but more importantly, it has led to erroneous comparisons and conclusions. In this paper, we first present a quick literature review of the recent advances in end-to-end RE. Our main contribution is then the identification of invalid comparison patterns in recent publications. We list them with the hope of stopping the propaga- tion of erroneous results and presenting a curated list of published results. To further this contribu- tion, we propose a small empirical study to quantify the impact of switching the two main metrics and estimate it can lead to a relative overestimation of around 5% in the end-to-end RE results on ACE05. As a second contribution, we take advantage of this quantitative study to perform the omitted abla- tions of two recent developments in the literature: LM pretraining and Span-level NER. It confirms that recent empirical gains are mainly due to LM pretraining, while there is no evidence for quantita- tive gains from …

Content truncated to 500 words