Consensus Maximization Fusion of Probabilistic Information Extractors

Proceedings of NAACL-HLT 2016, pages 1208–1216, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Consensus Maximization Fusion of Probabilistic Information Extractors Miguel Rodr´ıguez and Sean Goldberg and Daisy Zhe Wang University of Florida, Dept of Computer Science, Gainesville, FL, USA {mer,sean,daisyw}@cise.ufl.edu Abstract Current approaches to Information Extraction (IE) are capable of extracting large amounts of facts with associated probabilities. Because no current IE system is perfect, complemen- tary and conflicting facts are obtained when different systems are run over the same data. Knowledge Fusion (KF) is the problem of ag- gregating facts from different extractors. Ex- isting methods approach KF using supervised learning or deep linguistic knowledge, which either lack sufficient data or are not robust enough. We propose a semi-supervised appli- cation of Consensus Maximization to the KF problem, using a combination of supervised and unsupervised models. Consensus Maxi- mization Fusion (CM Fusion) is able to pro- mote high quality facts and eliminate incor- rect ones. We demonstrate the effectiveness of our system on the NIST Slot Filler Valida- tion contest, which seeks to evaluate and ag- gregate multiple independent information ex- tractors. Our system achieved the highest F1 score relative to other system submissions. 1 Introduction The abundance of unstructured text on the web such as news, discussion forums, wiki pages, etc. has in- creased the interest of the research community in the automatic extraction of information at scale. Infor- mation extractors can be used to construct or expand Knowledge Bases (KBs) through a process known as Knowledge Base Population (KBP) or Construction. Facts in a KB are typically modeled as (subject, relation, object) triples such as (Facebook, org:city of headquarters, Menlo Park). No current information extractor is perfectly accurate and different models exhibit different strengths and weaknesses. As a result, many state- of-the-art KBs in academia and industry employ multiple complementary information extractors for KBP. NELL(Mitchell and Fredkin, 2014) employs rules, statistically-learned pattern extractors, and context extractors among others. YAGO (Suchanek et al., 2007) uses heuristic extractors at text and on- tological levels and Google has multiple extractors crawling text, tables, and HTML. Different extraction systems may also agree or disagree on the information they extract. Con- sider three systems that extract the facts (Facebook, org:city of headquarters, Menlo Park), (Facebook, org:city of headquarters, Palo Alto), and (Face- book, org:city of headquarters, Menlo Park) with probabilities 0.6, 0.3, and 0.5 respectively. The Palo Alto extraction is erroneous and should be removed. The two Menlo Park extractions should be promoted by agreement and have their confidences increased. The aggregation of facts from multiple extractors into a single probabilistic triple is known as Knowl- edge Fusion (KF) (Dong et al., 2014) and can be modeled as an ensemble learning problem. Previous ensemble approaches at the output layer divide into unsupervised and supervised methods (Gao et al., 2010). Unsupervised methods establish a consen- sus or majority vote among extractors without distin- guishing the merit of each, but perform poorly if all the extractors are weak. Supervised methods …

Content truncated to 500 words