Confidence Estimation for Knowledge Base Population

Proceedings of Recent Advances in Natural Language Processing, pages 396–401, Hissar, Bulgaria, 7-13 September 2013. Confidence Estimation for Knowledge Base Population Xiang Li Computer Science Department New York University xiangli@cs.nyu.edu Ralph Grishman Computer Science Department New York University grishman@cs.nyu.edu Abstract Information extraction systems automati- cally extract structured information from machine-readable documents, such as newswire, web, and multimedia. Despite significant improvement, the performance is far from perfect. Hence, it is useful to accurately estimate confidence in the cor- rectness of the extracted information. Us- ing the Knowledge Base Population Slot Filling task as a case study, we propose a confidence estimation model based on the Maximum Entropy framework, obtaining an average precision of 83.5%, Pearson coefficient of 54.2%, and 2.3% absolute improvement in F-measure score through a weighted voting strategy. 1 Introduction Despite significant progress in recent years, In- formation Extraction (IE) technologies are still far from completely reliable. Errors result from the fact that language itself is ambiguous as well as methodological and technical limitations (Gan- drabur et al., 2006). Therefore, evaluating the probability that the extracted information is cor- rect can contribute to improve IE system perfor- mance. Confidence Estimation (CE) is a generic machine learning rescoring approach for measur- ing the probability of correctness of the outputs, and usually adds a layer on top of the baseline sys- tem to analyze the outputs using additional infor- mation or models (Gandrabur et al., 2006). There is previous work in IE using probabilistic and heuristic methods to estimate confidence for ex- tracting fields using a sequential model, but to the best of our knowledge, this work is the first proba- bilistic CE model for the multi-stage systems em- ployed for the Knowledge Base Population (KBP) Slot Filling task (Section 2). The goal of Slot Filling (SF) is to collect infor- mation from a corpus of news and web documents to determine a set of predefined attributes (“slots”) for given person and organization entities (Ji et al., 2011a) (Section 3). Many existing methodologies have been used to address the SF task, such as Dis- tant Supervision (Min et al., 2012) and Question Answering (Chen et al., 2010), and each method has its own strengths and weaknesses. Many cur- rent KBP SF systems actually consist of several independent SF pipelines. The system combines intermediate responses generated from different pipelines into final slot fills. Since these interme- diate outputs may be highly redundant, if confi- dence values can be associated, it will definitely help re-ranking and aggregation. For this pur- pose, we require comparable confidence values from disparate machine learning models or differ- ent slot filling strategies. Robust probabilistic machine learning models are capable of accurate confidence estimation be- cause of their intelligent handling of uncertainty information. In this paper, we use the Maximum Entropy (MaxEnt) framework (Berger et al., 1996) to automatically predict the correctness of KBP SF intermediate responses (Section 4). Results achieve an average precision of 83.5%, Pearson’s r of 54.2%, and 2.3% absolute improvement in final F-measure score through …

Content truncated to 500 words