Importance sampling for unbiased on-demand evaluation of knowledge base population

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1038–1048 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Importance sampling for unbiased on-demand evaluation of knowledge base population Arun Tejasvi Chaganty∗ and Ashwin Pradeep Paranjape∗and Percy Liang Computer Science Department Stanford University {chaganty,ashiwnp,pliang}@cs.stanford.edu Christopher D. Manning Computer Science Department Stanford University {manning}@cs.stanford.edu Abstract Knowledge base population (KBP) sys- tems take in a large document corpus and extract entities and their relations. Thus far, KBP evaluation has relied on judge- ments on the pooled predictions of exist- ing systems. We show that this evalua- tion is problematic: when a new system predicts a previously unseen relation, it is penalized even if it is correct. This leads to significant bias against new systems, which counterproductively discourages in- novation in the field. Our first contribu- tion is a new importance-sampling based evaluation which corrects for this bias by annotating a new system’s predictions on- demand via crowdsourcing. We show this eliminates bias and reduces variance using data from the 2015 TAC KBP task. Our second contribution is an implementation of our method made publicly available as an online KBP evaluation service. We pi- lot the service by testing diverse state-of- the-art systems on the TAC KBP 2016 cor- pus and obtain accurate scores in a cost ef- fective manner. 1 Introduction Harnessing the wealth of information present in unstructured text online has been a long stand- ing goal for the natural language processing com- munity. In particular, knowledge base popula- tion seeks to automatically construct a knowl- edge base consisting of relations between entities from a document corpus. Knowledge bases have found many applications including question an- swering (Berant et al., 2013; Fader et al., 2014; ∗ Authors contributed equally. Fisher’s mother, entertainer Debbie Reynolds, said on Twitter on Sunday that her daughter was stabilizing. Debbie Reynolds , title, entertainer Debbie Reynolds, per:parents, Fisher Fisher Debbie Reynolds daughter her Twitter Relation instances Linked Entities Knowledge Base title entertainer child of + Figure 1: An example describing entities and re- lations in knowledge base population. Reddy et al., 2014), automated reasoning (Kalyan- pur et al., 2012) and dialogue (Han et al., 2015). Evaluating these systems remains a challenge as it is not economically feasible to exhaustively annotate every possible candidate relation from a sufficiently large corpus. As a result, a pooling- based methodology is used in practice to construct datasets, similar to them methodology used in in- formation retrieval (Jones and Rijsbergen, 1975; Harman, 1993). For instance, at the annual NIST TAC KBP evaluation, all relations predicted by participating systems are pooled together, anno- tated and released as a dataset for researchers to develop and evaluate their systems on. However, during development, if a new system predicts a previously unseen relation it is considered to be wrong even if it is correct. The discrepancy be- tween a system’s true score and the score on the pooled dataset is called pooling bias and is typi- cally assumed to be …

Content truncated to 500 words