Improved Extraction Assessment through Better Language Models

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 225–228, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics Improved Extraction Assessment through Better Language Models Arun Ahuja, Doug Downey EECS Dept., Northwestern University Evanston, IL 60208 {arun.ahuja, ddowney}@eecs.northwestern.edu Abstract A variety of information extraction techniques rely on the fact that instances of the same relation are “distributionally similar,” in that they tend to appear in similar textual con- texts. We demonstrate that extraction accu- racy depends heavily on the accuracy of the language model utilized to estimate distribu- tional similarity. An unsupervised model se- lection technique based on this observation is shown to reduce extraction and type-checking error by 26% over previous results, in experi- ments with Hidden Markov Models. The re- sults suggest that optimizing statistical lan- guage models over unlabeled data is a promis- ing direction for improving weakly supervised and unsupervised information extraction. 1 Introduction Many weakly supervised and unsupervised informa- tion extraction techniques assess the correctness of extractions using the distributional hypothesis—the notion that words with similar meanings tend to oc- cur in similar contexts (Harris, 1985). A candidate extraction of a relation is deemed more likely to be correct when it appears in contexts similar to those of “seed” instances of the relation, where the seeds may be specified by hand (Pas¸ca et al., 2006), taken from an existing, incomplete knowledge base (Snow et al., 2006; Pantel et al., 2009), or obtained in an un- supervised manner using a generic extractor (Banko et al., 2007). We refer to this technique as Assess- ment by Distributional Similarity (ADS). Typically, distributional similarity is computed by comparing co-occurrence counts of extractions and seeds with various contexts found in the corpus. Sta- tistical Language Models (SLMs) include methods for more accurately estimating co-occurrence proba- bilities via back-off, smoothing, and clustering tech- niques (e.g. (Chen and Goodman, 1996; Rabiner, 1989; Bell et al., 1990)). Because SLMs can be trained from only unlabeled text, they can be applied for ADS even when the relations of interest are not specified in advance (Downey et al., 2007). Unla- beled text is abundant in large corpora like the Web, making nearly-ceaseless automated optimization of SLMs possible. But how fruitful is such an effort likely to be—to what extent does optimizing a lan- guage model over a fixed corpus lead to improve- ments in assessment accuracy? In this paper, we show that an ADS technique based on SLMs is improved substantially when the language model it employs becomes more ac- curate. In a large-scale set of experiments, we quantify how language model perplexity correlates with ADS performance over multiple data sets and SLM techniques. The experiments show that accu- racy over unlabeled data can be used for selecting among SLMs—for an ADS approach utilizing Hid- den Markov Models, this results in an average error reduction of 26% over previous results in extraction and type-checking tasks. 2 Extraction Assessment with Language Models We begin by formally defining …

Content truncated to 500 words