The Wisdom of Minority: Unsupervised Slot Filling Validation based on Multi-dimensional Truth-Finding

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1567–1578, Dublin, Ireland, August 23-29 2014. The Wisdom of Minority: Unsupervised Slot Filling Validation based on Multi-dimensional Truth-Finding Dian Yu1, Hongzhao Huang1, Taylor Cassidy2,3, Heng Ji1 Chi Wang4, Shi Zhi4, Jiawei Han4, Clare Voss2, Malik Magdon-Ismail1 1Computer Science Department, Rensselaer Polytechnic Institute 2U.S. Army Research Lab 3IBM T. J. Watson Research Center 4Computer Science Department, Univerisity of Illinois at Urbana-Champaign 1{yud2,huangh9,jih,magdon}@rpi.edu, 2,3{taylor.cassidy.ctr,clare.r.voss.civ}@mail.mil 4{chiwang1,shizhi2,hanj}@illinois.edu Abstract Information Extraction using multiple information sources and systems is beneficial due to multi- source/system consolidation and challenging due to the resulting inconsistency and redundancy. We integrate IE and truth-finding research and present a novel unsupervised multi-dimensional truth finding framework which incorporates signals from multiple sources, multiple systems and multiple pieces of evidence by knowledge graph construction through multi-layer deep linguistic analysis. Experiments on the case study of Slot Filling Validation demonstrate that our approach can find truths accurately (9.4% higher F-score than supervised methods) and efficiently (finding 90% truths with only one half the cost of a baseline without credibility estimation). 1 Introduction Traditional Information Extraction (IE) techniques assess the ability to extract information from individual documents in isolation. However, similar, complementary or conflicting information may exist in multiple heterogeneous sources. We take the Slot Filling Validation (SFV) task of the NIST Text Analysis Conference Knowledge Base Population (TAC-KBP) track (Ji et al., 2011) as a case study. The Slot Filling (SF) task aims at collecting from a large-scale multi-source corpus the values (“slot fillers”) for certain attributes (“slot types”) of a query entity, which is a person or some type of organization. KBP 2013 has defined 25 slot types for persons (per) (e.g., age, spouse, employing organization) and 16 slot types for organizations (org) (e.g., founder, headquarters-location, and subsidiaries). Some slot types take only a single slot filler (e.g., per:birthplace), whereas others take multiple slot fillers (e.g., org:top employees). We call a combination of query entity, slot type, and slot filler a claim. Along with each claim, each system must provide the ID of a source document and one or more detailed context sentences as evidence which supports the claim. A response (i.e., a claim, evidence pair) is correct if and only if the claim is true and the evidence supports it. Given the responses produced by multiple systems from multiple sources in the SF task, the goal of the SFV task is to determine whether each response is true or false. Though it’s a promising line of research, it raises two complications: (1) different information sources may generate claims that vary This work is licenced under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organizers. License details: http://creativecommons.org/licenses/by/4.0/ 1567 in trustability; and (2) a large-scale number of SF systems using different resources and algorithms may generate erroneous, conflicting, redundant, complementary, ambiguously worded, or inter-dependent claims from the same set of documents. Table 1 presents responses from four SF systems for the query …

Content truncated to 500 words