When ACE met KBP: End-to-End Evaluation of Knowledge Base Population with Component-level Annotation

When ACE met KBP1: End-to-End Evaluation of Knowledge Base Population with Component-level Annotation Bonan Min†, Marjorie Freedman‡, Roger Bock†, Ralph Weischedel‡ † Raytheon BBN Technologies, Cambridge, MA 02138 {bonan.min, roger.bock}@raytheon.com ‡ USC/Information Sciences Institute, Marina del Rey, CA 90292 {mrf, weisched}@isi.edu Abstract Building a Knowledge Base from text corpora is useful for many applications such as question answering and web search. Since 2012, the Cold Start Knowledge Base Population (KBP) evaluation at the Text Analysis Conference (TAC) has attracted many participants. Despite the popularity, the Cold Start KBP evaluation has several problems including but not limited to the following two: first, each year’s assessment dataset is a pooled set of query-answer pairs, primarily generated by participating systems. It is well known to participants that there is pooling bias: a system developed outside of the official evaluation period is not rewarded for finding novel answers, but rather is penalized for doing so. Second, the assessment dataset, constructed with lots of human effort, offers little help in training information extraction algorithms which are crucial ingredients for the end-to-end KBP task. To address these problems, we propose a new unbiased evaluation methodology that uses existing component-level annotation such as the Automatic Content Extraction (ACE) dataset, to evaluate Cold Start KBP. We also propose bootstrap resampling to provide statistical significance to the results reported. We will then present experimental results and analysis. Keywords: Information Extraction, Knowledge Base Population, evaluation 1. Introduction Automatically constructing a Knowledge Base (KB) of entities and relations from unstructured text, has long been a goal of Natural Language Processing (NLP). The task, named Knowledge Based Population (KBP), will unlock the huge potential in unstructured text for applications such as questions answering and web search. 1 Since 2012, the National Institute of Standards and Technology (NIST) has run the TAC Cold Start KBP evaluation, which measures performance of KBP. As the successor to the Message Understanding Conference (MUC) (Grishman and Sundheim, 1996) and Automatic Content Extraction (ACE) (Doddington et al., 2004) evaluations, Cold Start KBP evaluates a system’s ability to automatically construct a KB from text. It uses a large corpus of 50,000-90,000 documents which have not gone through a careful selection process. In Cold Start KBP evaluation, a system is required to submit a KB 2 of entities and relations, constructed automatically from the corpus by algorithms. How can one evaluate the quality of a KB? The Cold Start KBP evaluation2 measures it by probing the KB with two types of queries: 1-hop (e.g., which organization(s) is(are) founded by Bill Gates?) or 2-hop (e.g., in which city(-ies) is(are) the organization(s) founded by Bill Gates headquartered?). The evaluation software traverses a KB and finds all answers to the 1-hop and 2-hop queries. Human annotators then annotate the correctness of a system answer by checking whether it is sufficiently 1 Inspired by the movie “When Harry Met Sally…” in which two friends with drastically different personalities found each other to be the love of their life. 2 The schema of the KB …

Content truncated to 500 words