Speed and Accuracy in Shallow and Deep Stochastic Parsing

Speed and Accuracy in Shallow and Deep Stochastic Parsing Ronald M. Kaplan , Stefan Riezler , Tracy Holloway King John T. Maxwell III, Alexander Vasserman and Richard Crouch Palo Alto Research Center, 3333 Coyote Hill Rd., Palo Alto, CA 94304 {kaplan|riezler|king|maxwell|avasserm|crouch}@parc.com Abstract This paper reports some experiments that com- pare the accuracy and performance of two stochastic parsing systems. The currently pop- ular Collins parser is a shallow parser whose output contains more detailed semantically- relevant information than other such parsers. The XLE parser is a deep-parsing system that couples a Lexical Functional Grammar to a log- linear disambiguation component and provides much richer representations theory. We mea- sured the accuracy of both systems against a gold standard of the PARC 700 dependency bank, and also measured their processing times. We found the deep-parsing system to be more accurate than the Collins parser with only a slight reduction in parsing speed.1 1 Introduction In applications that are sensitive to the meanings ex- pressed by natural language sentences, it has become common in recent years simply to incorporate publicly available statistical parsers. A state-of-the-art statistical parsing system that enjoys great popularity in research systems is the parser described in Collins (1999) (hence- forth “the Collins parser”). This system not only is fre- quently used for off-line data preprocessing, but also is included as a black-box component for applications such as document summarization (Daume and Marcu, 2002), information extraction (Miller et al., 2000), ma- chine translation (Yamada and Knight, 2001), and ques- tion answering (Harabagiu et al., 2001). This is be- 1This research has been funded in part by contract # MDA904-03-C-0404 awarded from the Advanced Research and Development Activity, Novel Intelligence from Massive Data program. We would like to thank Chris Culy whose original ex- periments inspired this research. cause the Collins parser shares the property of robustness with other statistical parsers, but more than other such parsers, the categories of its parse-trees make grammati- cal distinctions that presumably are useful for meaning- sensitive applications. For example, the categories of the Model 3 Collins parser distinguish between heads, arguments, and adjuncts and they mark some long- distance dependency paths; these distinctions can guide application-specific postprocessors in extracting impor- tant semantic relations. In contrast, state-of-the-art parsing systems based on deep grammars mark explicitly and in much more de- tail a wider variety of syntactic and semantic dependen- cies and should therefore provide even better support for meaning-sensitive applications. But common wisdom has it that parsing systems based on deep linguistic grammars are too difficult to produce, lack coverage and robustness, and also have poor run-time performance. The Collins parser is thought to be accurate and fast and thus to repre- sent a reasonable trade-off between “good-enough” out- put, speed, and robustness. This paper reports on some experiments that put this conventional wisdom to an empirical test. We investi- gated the accuracy of recovering semantically-relevant grammatical dependencies from the tree-structures pro- duced by the Collins parser, comparing these dependen- cies to gold-standard dependencies …

Content truncated to 500 words