Detecting Parser Errors Using Web-based Semantic Filters

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 27–34, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Detecting Parser Errors Using Web-based Semantic Filters Alexander Yates Stefan Schoenmackers University of Washington Computer Science and Engineering Box 352350 Seattle, WA 98195-2350 {ayates, stef, etzioni} @cs.washington.edu Oren Etzioni Abstract NLP systems for tasks such as question answering and information extraction typ- ically rely on statistical parsers. But the ef- ficacy of such parsers can be surprisingly low, particularly for sentences drawn from heterogeneous corpora such as the Web. We have observed that incorrect parses of- ten result in wildly implausible semantic interpretations of sentences, which can be detected automatically using semantic in- formation obtained from the Web. Based on this observation, we introduce Web-based semantic filtering—a novel, domain-independent method for automat- ically detecting and discarding incorrect parses. We measure the effectiveness of our filtering system, called WOODWARD, on two test collections. On a set of TREC questions, it reduces error by 67%. On a set of more complex Penn Treebank sentences, the reduction in error rate was 20%. 1 Introduction Semantic processing of text in applications such as question answering or information extraction frequently relies on statistical parsers. Unfortu- nately, the efficacy of state-of-the-art parsers can be disappointingly low. For example, we found that the Collins parser correctly parsed just 42% of the list and factoid questions from TREC 2004 (that is, 42% of the parses had 100% precision and 100% recall on labeled constituents). Similarly, this parser produced 45% correct parses on a sub- set of 100 sentences from section 23 of the Penn Treebank. Although statistical parsers continue to improve their efficacy over time, progress is slow, par- ticularly for Web applications where training the parsers on a “representative” corpus of hand- tagged sentences is not an option. Because of the heterogeneous nature of text on the Web, such a corpus would be exceedingly difficult to generate. In response, this paper investigates the possibil- ity of detecting parser errors by using semantic in- formation obtained from the Web. Our fundamen- tal hypothesis is that incorrect parses often result in wildly implausible semantic interpretations of sentences, which can be detected automatically in certain circumstances. Consider, for example, the following sentence from the Wall Street Journal: “That compares with per-share earnings from con- tinuing operations of 69 cents.” The Collins parser yields a parse that attaches “of 69 cents” to “op- erations,” rather than “earnings.” By computing the mutual information between “operations” and “cents” on the Web, we can detect that this attach- ment is unlikely to be correct. Our WOODWARD system detects parser errors as follows. First, it maps the tree produced by a parser to a relational conjunction (RC), a logic- based representation language that we describe in Section 2.1. Second, WOODWARD employs four distinct methods for analyzing whether a conjunct in the RC is likely to be “reasonable” as described in Section 2. Our approach makes several assumptions. First, if the sentence …

Content truncated to 500 words