Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10438–10449 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10438 Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction Bruno Taillé1, Vincent Guigue1, Geoffrey Scoutheeten2 and Patrick Gallinari1,3 1Sorbonne Université, CNRS, Laboratoire d’Informatique de Paris 6, LIP6 2BNP Paribas 3Criteo AI Lab {bruno.taille, vincent.guigue, patrick.gallinari}@lip6.fr geoffrey.scoutheeten@bnpparibas.com Abstract State-of-the-art NLP models can adopt shal- low heuristics that limit their generalization ca- pability (McCoy et al., 2019). Such heuris- tics include lexical overlap with the training set in Named-Entity Recognition (Taillé et al., 2020a) and Event or Type heuristics in Rela- tion Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere re- tention of training relation triples. In this pa- per we propose several experiments confirm- ing that retention of known facts is a key fac- tor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on re- tention. 1 Introduction Information Extraction (IE) aims at converting the information expressed in a text into a predefined structured format of knowledge. This global goal has been divided into subtasks easier to perform automatically and evaluate. Hence, Named Entity Recognition (NER) and Relation Extraction (RE) are two key IE tasks among others such as Coref- erence Resolution (CR), Entity Linking or Event Extraction. Traditionally performed as a pipeline (Bach and Badaskar, 2007), these two tasks can be tackled jointly in order to model their interde- pendency, alleviate error propagation and obtain a more realistic evaluation setting (Roth and Yih, 2002; Li and Ji, 2014). Following the general trend in Natural Language Processing (NLP), the recent quantitative improve- ments reported on Entity and Relation Extraction benchmarks are at least partly explained by the use of larger and larger pretrained Language Models (LMs) such as BERT (Devlin et al., 2019) to ob- tain contextual word representations. Concurrently, Code for reproducing our evaluation settings is available at github.com/btaille/retex there is a realization that new evaluation protocols are necessary to better understand the strengths and shortcomings of the obtained neural network mod- els, beyond a single holistic metric on an hold-out test set (Ribeiro et al., 2020). In particular, generalisation to unseen data is a key factor in the evaluation of deep neural networks. It is all the more important in IE tasks that revolve around the extraction of mentions: small spans of words that are likely to occur in both the evaluation and training datasets. This lexical overlap has been shown to be correlated to neural networks perfor- mance in NER (Augenstein et al., 2017; Taillé et al., 2020a). For pipeline RE, Rosenman et al. (2020) and Peng et al. (2020) expose shallow heuristics in neural models: relying too much on the type of the candidate arguments or on the presence of specific triggers in their contexts. In end-to-end Relation Extraction, we can expect that these …

Content truncated to 500 words