Predicting Document Coverage for Relation Extraction

Predicting Document Coverage for Relation Extraction Sneha Singhania, Simon Razniewski, Gerhard Weikum Max Planck Institute for Informatics, Germany {ssinghan,srazniew,weikum}@mpi-inf.mpg.de Abstract This paper presents a new task of predicting the coverage of a text document for relation extraction (RE): Does the document contain many relational tuples for a given entity? Coverage predictions are useful in selecting the best documents for knowledge base con- struction with large input corpora. To study this problem, we present a dataset of 31,366 diverse documents for 520 entities. We an- alyze the correlation of document coverage with features like length, entity mention fre- quency, Alexa rank, language complexity, and information retrieval scores. Each of these features has only moderate predictive power. We employ methods combining features with statistical models like TF-IDF and language models like BERT. The model combining fea- tures and BERT, HERB, achieves an F1 score of up to 46%. We demonstrate the utility of coverage predictions on two use cases: KB construction and claim refutation. 1 Introduction Motivation and Problem Relation extraction (RE) from text documents is an important NLP task with a range of downstream applications (Han et al., 2020). For these applications, it is vital to understand the quality of RE results. While extractors typically provide confidence (or preci- sion) scores, this paper puts forward the notion of RE coverage (or recall). Given an input doc- ument and an RE method, coverage measures the fraction of the extracted relations compared to the complete ground-truth that holds in real- ity. We consider this on a per-subject and per- predicate basis—for example, all memberships of Bill Gates in organizations or all companies founded by Elon Musk. Document coverage for RE highly varies. Con- sider the three text snippets about Tesla as shown in Figure 1. The first text contains all five founders of Tesla, while the second text contains only two of them, and the third has just one. Analogously, for the entity Tesla and the relation founded-by, we see that text 1 has coverage 1, text 2 has coverage 0.4, and text 3 has coverage 0.2. When applying RE at scale, for example, to populate or augment a knowledge base (KB), an RE system may have to process a huge number of input documents that differ widely in their coverage. As state-of-the-art extractors are based on heavy-duty neural networks (Lin et al., 2016; Zhang et al., 2017; Soares et al., 2019; Yao et al., 2019), processing all documents in a large cor- pus may be prohibitive and inefficient. Instead, prioritizing the input documents by identifying the best documents with high coverage could be effective. This is why coverage prediction is cru- cial for large-scale RE, but the problem has not been explored so far. This problem would be easy if we could first run a neural RE system on each document and then assess the yield, either by comparison to withheld labeled data or by sampling followed by human inspection. However, this is exactly the computational bottleneck that we must avoid. The …

Content truncated to 500 words