Task-oriented Evaluation of Syntactic Parsers and Their Representations

Proceedings of ACL-08: HLT, pages 46–54, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Task-oriented Evaluation of Syntactic Parsers and Their Representations Yusuke Miyao† Rune Sætre† Kenji Sagae† Takuya Matsuzaki† Jun’ichi Tsujii†‡∗ †Department of Computer Science, University of Tokyo, Japan ‡School of Computer Science, University of Manchester, UK ∗National Center for Text Mining, UK {yusuke,rune.saetre,sagae,matuzaki,tsujii}@is.s.u-tokyo.ac.jp Abstract This paper presents a comparative evalua- tion of several state-of-the-art English parsers based on different frameworks. Our approach is to measure the impact of each parser when it is used as a component of an information ex- traction system that performs protein-protein interaction (PPI) identification in biomedical papers. We evaluate eight parsers (based on dependency parsing, phrase structure parsing, or deep parsing) using five different parse rep- resentations. We run a PPI system with several combinations of parser and parse representa- tion, and examine their impact on PPI identi- fication accuracy. Our experiments show that the levels of accuracy obtained with these dif- ferent parsers are similar, but that accuracy improvements vary when the parsers are re- trained with domain-specific data. 1 Introduction Parsing technologies have improved considerably in the past few years, and high-performance syntactic parsers are no longer limited to PCFG-based frame- works (Charniak, 2000; Klein and Manning, 2003; Charniak and Johnson, 2005; Petrov and Klein, 2007), but also include dependency parsers (Mc- Donald and Pereira, 2006; Nivre and Nilsson, 2005; Sagae and Tsujii, 2007) and deep parsers (Kaplan et al., 2004; Clark and Curran, 2004; Miyao and Tsujii, 2008). However, efforts to perform extensive comparisons of syntactic parsers based on different frameworks have been limited. The most popular method for parser comparison involves the direct measurement of the parser output accuracy in terms of metrics such as bracketing precision and recall, or dependency accuracy. This assumes the existence of a gold-standard test corpus, such as the Penn Tree- bank (Marcus et al., 1994). It is difficult to apply this method to compare parsers based on different frameworks, because parse representations are often framework-specific and differ from parser to parser (Ringger et al., 2004). The lack of such comparisons is a serious obstacle for NLP researchers in choosing an appropriate parser for their purposes. In this paper, we present a comparative evalua- tion of syntactic parsers and their output represen- tations based on different frameworks: dependency parsing, phrase structure parsing, and deep pars- ing. Our approach to parser evaluation is to mea- sure accuracy improvement in the task of identify- ing protein-protein interaction (PPI) information in biomedical papers, by incorporating the output of different parsers as statistical features in a machine learning classifier (Yakushiji et al., 2005; Katrenko and Adriaans, 2006; Erkan et al., 2007; Sætre et al., 2007). PPI identification is a reasonable task for parser evaluation, because it is a typical information extraction (IE) application, and because recent stud- ies have shown the effectiveness of syntactic parsing in this task. Since our evaluation method is applica- ble to any parser output, and is grounded in a real application, …

Content truncated to 500 words