Sparse Information Extraction: Unsupervised Language Models to the Rescue

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 696–703, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Sparse Information Extraction: Unsupervised Language Models to the Rescue Doug Downey, Stefan Schoenmackers, and Oren Etzioni Turing Center, Department of Computer Science and Engineering University of Washington, Box 352350 Seattle, WA 98195, USA {ddowney,stef,etzioni}@cs.washington.edu Abstract Even in a massive corpus such as the Web, a substantial fraction of extractions appear in- frequently. This paper shows how to assess the correctness of sparse extractions by uti- lizing unsupervised language models. The REALM system, which combines HMM- based and n-gram-based language models, ranks candidate extractions by the likeli- hood that they are correct. Our experiments show that REALM reduces extraction error by 39%, on average, when compared with previous work. Because REALM pre-computes language models based on its corpus and does not re- quire any hand-tagged seeds, it is far more scalable than approaches that learn mod- els for each individual relation from hand- tagged data. Thus, REALM is ideally suited for open information extraction where the relations of interest are not specified in ad- vance and their number is potentially vast. 1 Introduction Information Extraction (IE) from text is far from in- fallible. In response, researchers have begun to ex- ploit the redundancy in massive corpora such as the Web in order to assess the veracity of extractions (e.g., (Downey et al., 2005; Etzioni et al., 2005; Feldman et al., 2006)). In essence, such methods uti- lize extraction patterns to generate candidate extrac- tions (e.g., “Istanbul”) and then assess each candi- date by computing co-occurrence statistics between the extraction and words or phrases indicative of class membership (e.g., “cities such as”). However, Zipf’s Law governs the distribution of extractions. Thus, even the Web has limited redun- dancy for less prominent instances of relations. In- deed, 50% of the extractions in the data sets em- ployed by (Downey et al., 2005) appeared only once. As a result, Downey et al.’s model, and re- lated methods, had no way of assessing which ex- traction is more likely to be correct for fully half of the extractions. This problem is particularly acute when moving beyond unary relations. We refer to this challenge as the task of assessing sparse extrac- tions. This paper introduces the idea that language mod- eling techniques such as n-gram statistics (Manning and Sch¨utze, 1999) and HMMs (Rabiner, 1989) can be used to effectively assess sparse extractions. The paper introduces the REALM system, and highlights its unique properties. Notably, REALM does not require any hand-tagged seeds, which enables it to scale to Open IE—extraction where the relations of interest are not specified in advance, and their num- ber is potentially vast (Banko et al., 2007). REALM is based on two key hypotheses. The KnowItAll hypothesis is that extractions that oc- cur more frequently in distinct sentences in the corpus are more likely to be correct. For exam- ple, the hypothesis suggests that the argument pair (Giuliani, New …

Content truncated to 500 words