TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558–1569 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1558 TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task Christoph Alt Aleksandra Gabryszak Leonhard Hennig German Research Center for Artificial Intelligence (DFKI) Speech and Language Technology Lab {christoph.alt, aleksandra.gabryszak, leonhard.hennig}@dfki.de Abstract TACRED (Zhang et al., 2017) is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pre- training and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K exam- ples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be rela- beled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we ana- lyze misclassifications on the challenging in- stances, categorize them into linguistically mo- tivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambigu- ous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked. 1 Introduction Relation Extraction (RE) is the task of extract- ing relationships between concepts and entities from text, where relations correspond to seman- tic categories such as per:spouse, org:founded by or org:subsidiaries (Figure 1). This makes RE a key part of many information extraction systems, and its performance determines the quality of ex- tracted facts for knowledge base population (Ji and Grishman, 2011), or the quality of answers in ques- tion answering systems (Xu et al., 2016). Standard benchmarks such as SemEval 2010 Task 8 (Hen- drickx et al., 2010) and the more recent TACRED [...] included Aerolineas’s domestic subsidiary, Austral. org:subsidiaries tail (obj) head (subj) Figure 1: Relation example from TACRED. The sen- tence contains the relation org:subsidiaries between the head and tail organization entities ‘Aerolineas’ and ‘Austral’. (Zhang et al., 2017) are essential to evaluate new RE methods and their limitations, and to establish open challenges. TACRED is one of the largest and most widely used RE datasets. It contains more than 106k ex- amples annotated by crowd workers. The meth- ods best performing on the dataset use some form of pre-training to improve RE performance: fine- tuning pre-trained language representations (Alt et al., 2019; Shi and Lin, 2019; Joshi et al., 2019) or integrating external knowledge during pre-training, e.g. via joint language modelling and linking on entity-linked text (Zhang et al., 2019; Peters et al., 2019; Baldini Soares et al., 2019); with the last two methods achieving …

Content truncated to 500 words