Modeling Missing Data in Distant Supervision for Information Extraction

Modeling Missing Data in Distant Supervision for Information Extraction Alan Ritter Machine Learning Department Carnegie Mellon University rittera@cs.cmu.edu Luke Zettlemoyer, Mausam Computer Sci. & Eng. University of Washington {lsz,mausam}@cs.washington.edu Oren Etzioni Vulcan Inc. Seattle, WA orene@vulcan.com Abstract Distant supervision algorithms learn informa- tion extraction models given only large read- ily available databases and text collections. Most previous work has used heuristics for generating labeled data, for example assum- ing that facts not contained in the database are not mentioned in the text, and facts in the database must be mentioned at least once. In this paper, we propose a new latent-variable approach that models missing data. This pro- vides a natural way to incorporate side in- formation, for instance modeling the intuition that text will often mention rare entities which are likely to be missing in the database. De- spite the added complexity introduced by rea- soning about missing data, we demonstrate that a carefully designed local search approach to inference is very accurate and scales to large datasets. Experiments demonstrate im- proved performance for binary and unary re- lation extraction when compared to learning with heuristic labels, including on average a 27% increase in area under the precision re- call curve in the binary case. 1 Introduction This paper addresses the issue of missing data (Lit- tle and Rubin, 1986) in the context of distant super- vision. The goal of distant supervision is to learn to process unstructured data, for instance to extract binary or unary relations from text (Bunescu and Mooney, 2007; Snyder and Barzilay, 2007; Wu and Weld, 2007; Mintz et al., 2009; Collins and Singer, 1999), using a large database of propositions as a Person EMPLOYER Bibb Latan´e UNC Chapel Hill Tim Cook Apple Susan Wojcicki Google True Positive “Bibb Latan´e, a professor at the University of North Carolina at Chapel Hill, published the theory in 1981.” False Positive “Tim Cook praised Apple’s record revenue...” False Negative “John P. McNamara, a professor at Washington State University’s Department of Animal Sciences...” Figure 1: A small hypothetical database and heuris- tically labeled training data for the EMPLOYER rela- tion. distant source of supervision. In the case of binary relations, the intuition is that any sentence which mentions a pair of entities (e1 and e2) that partici- pate in a relation, r, is likely to express the proposi- tion r(e1, e2), so we can treat it as a positive training example of r. Figure 1 presents an example of this process. One question which has received little attention in previous work is how to handle the situation where information is missing, either from the text corpus, or the database. As an example, suppose the pair of entities (John P. McNamara, Washington State Uni- versity) is absent from the EMPLOYER relation. In this case, the sentence in Figure 1 (and others which mention the entity pair) is effectively treated as a negative example of the relation. This is an issue 367 Transactions of the Association for Computational Linguistics, 1 (2013) …

Content truncated to 500 words