Comparing and Combining Finite-State and Context-Free Parsers

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 787–794, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Comparing and Combining Finite-State and Context-Free Parsers Kristy Hollingshead and Seeger Fisher and Brian Roark Center for Spoken Language Understanding OGI School of Science & Engineering Oregon Health & Science University Beaverton, Oregon, 97006 {hollingk,fishers,roark}@cslu.ogi.edu Abstract In this paper, we look at comparing high- accuracy context-free parsers with high- accuracy finite-state (shallow) parsers on several shallow parsing tasks. We show that previously reported compar- isons greatly under-estimated the perfor- mance of context-free parsers for these tasks. We also demonstrate that context- free parsers can train effectively on rel- atively little training data, and are more robust to domain shift for shallow pars- ing tasks than has been previously re- ported. Finally, we establish that combin- ing the output of context-free and finite- state parsers gives much higher results than the previous-best published results, on several common tasks. While the efficiency benefit of finite-state models is inarguable, the results presented here show that the corresponding cost in accu- racy is higher than previously thought. 1 Introduction Finite-state parsing (also called chunking or shallow parsing) has typically been motivated as a fast first- pass for – or approximation to – more expensive context-free parsing (Abney, 1991; Ramshaw and Marcus, 1995; Abney, 1996). For many very-large- scale natural language processing tasks (e.g. open- domain question answering from the web), context- free parsing may be too expensive, whereas finite- state parsing is many orders of magnitude faster and can also provide very useful syntactic annotations for large amounts of text. For this reason, finite-state parsing (hereafter referred to as shallow parsing) has received increasing attention in recent years. In addition to the clear efficiency benefit of shallow parsing, Li and Roth (2001) have further claimed both an accuracy and a robustness benefit versus context-free parsing. The output of a context- free parser, such as that of Collins (1997) or Char- niak (2000), can be transformed into a sequence of shallow constituents for comparison with the output of a shallow parser. Li and Roth demonstrated that their shallow parser, trained to label shallow con- stituents along the lines of the well-known CoNLL- 2000 task (Sang and Buchholz, 2000), outperformed the Collins parser in correctly identifying these con- stituents in the Penn Wall Street Journal (WSJ) Tree- bank (Marcus et al., 1993). They argued that their superior performance was due to optimizing directly for the local sequence labeling objective, rather than for obtaining a hierarchical analysis over the entire string. They further showed that their shallow parser trained on the Penn WSJ Treebank did a far better job of annotating out-of-domain sentences (e.g. con- versational speech) than the Collins parser. This paper re-examines the comparison of shal- low parsers with context-free parsers, beginning with a critical examination of how their outputs are compared. We demonstrate that changes to the conversion routine, which take into account differ- ences between the original …

Content truncated to 500 words