Laying the Groundwork for Knowledge Base Population: Nine Years of Linguistic Resources for TAC KBP

Laying the Groundwork for Knowledge Base Population: Nine Years of Linguistic Resources for TAC KBP Jeremy Getman, Joe Ellis, Stephanie Strassel, Zhiyi Song, Jennifer Tracey Linguistic Data Consortium, University of Pennsylvania 3600 Market Street, Suite 810, Philadelphia, PA 19104 {jgetman, joellis, strassel, zhiyi, garjen}@ldc.upenn.edu Abstract Knowledge Base Population (KBP) is an evaluation series within the Text Analysis Conference (TAC) evaluation campaign conducted by the National Institute of Standards and Technology (NIST). Over the past nine years TAC KBP evaluations have targeted information extraction technologies for the population of knowledge bases comprised of entities, relations, and events. Linguistic Data Consortium (LDC) has supported TAC KBP since 2009, developing, maintaining, and distributing linguistic resources in three languages for seven distinct evaluation tracks. This paper describes LDC's resource creation efforts for the various KBP tracks, and highlights changes made over the years to support evolving evaluation requirements. Keywords: knowledge base population, information extraction, linguistic resources 1. Introduction Text Analysis Conference (TAC) an annual series of open technology evaluations organized by the National Institute of Standards and Technology (NIST). The Knowledge Base Population (KBP) evaluation track (McNamee et al. 2010) encourages the development of systems that can extract information from unstructured multilingual text and in order to populate an existing or emergent knowledge base. Since the start of TAC KBP in 2009 Linguistic Data Consortium (LDC) has provided linguistic resources by building labeled training and test sets and by assessing system results. Each year’s KBP evaluation comprises a number of component evaluation tasks. Over time the KBP evaluations, and by extension the data to support those evaluations, has moved in the direction of greater complexity, with more integration of individual evaluation tracks and data sets, a greater emphasis on multilingual data for all tracks, and an increasing focus on events (in addition to entities and slots). These developments culminated in 2017 with an expanded end-to-end Cold Start track that evaluated systems’ ability to combine the other KBP tracks and extract all entities, relations, sentiments, and events from a document collection, perform corpus-wide clustering of the extracted items, and build a structured knowledge base from scratch (NIST, 2017). In the sections that follow we describe the KBP corpora developed by LDC for each of the seven primary evaluation tracks, including source data, knowledge bases and annotation and assessment efforts between 2009- 2017. TAC KBP will continue in 2018 with new evaluation tracks. 2. Source Data and Knowledge Bases Source data for the KBP evaluations consists of English, Spanish and Chinese text in two primary genres: formal newswire (NW) and informal web text (primarily blogs and discussion forums) drawn from existing LDC collections as well as newly collected material. In 2009- 2015 we selected a separate set of documents for each evaluation track. Starting in 2016, the consolidation of evaluations into a smaller number of more complex and challenging tasks led to the use of a single set of test data shared by all tasks. This put additional demands on data selection given the need for …

Content truncated to 500 words