Evaluating the Effects of Treebank Size in a Practical Application for Parsing

Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 14–20, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Evaluating the Effects of Treebank Size in a Practical Application for Parsing Kenji Sagae1, Yusuke Miyao1, Rune Sætre1 and Jun'ichi Tsujii1,2,3 1Department of Computer Science, Univerisity of Tokyo, Japan 2School of Computer Science, University of Manchester 3National Center for Text Mining, Manchester, UK {sagae,yusuke,rune.saetre,tsujii@is.s.u-tokyo.ac.jp} Abstract Natural language processing modules such as part-of-speech taggers, named-entity recog- nizers and syntactic parsers are commonly evaluated in isolation, under the assumption that artificial evaluation metrics for individual parts are predictive of practical performance of more complex language technology sys- tems that perform practical tasks. Although this is an important issue in the design and en- gineering of systems that use natural language input, it is often unclear how the accuracy of an end-user application is affected by parame- ters that affect individual NLP modules. We explore this issue in the context of a specific task by examining the relationship between the accuracy of a syntactic parser and the overall performance of an information extrac- tion system for biomedical text that includes the parser as one of its components. We present an empirical investigation of the rela- tionship between factors that affect the accu- racy of syntactic analysis, and how the difference in parse accuracy affects the overall system. 1 Introduction Software systems that perform practical tasks with natural language input often include, in addition to task-specific components, a pipeline of basic natu- ral language processing modules, such as part-of- speech taggers, named-entity recognizers, syntactic parsers and semantic-role labelers. Although such building blocks of larger language technology so- lutions are usually carefully evaluated in isolation using standard test sets, the impact of improve- ments in each individual module on the overall performance of end-to-end systems is less well understood. While the effects of the amount of training data, search beam widths and various ma- chine learning frameworks have been explored in detail with respect to speed and accuracy in basic natural language processing tasks, how these trade- offs in individual modules affect the performance of the larger systems they compose is an issue that has received relatively little attention. This issue, however, is of great practical importance in the effective design and engineering of complex soft- ware systems that deal with natural language. In this paper we explore some of these issues empirically in an information extraction task in the biomedical domain, the identification of protein- protein interactions (PPI) mentioned in papers ab- stracts from MEDLINE, a large database of bio- medical papers. Due in large part to the creation of biomedical treebanks (Kulick et al., 2004; Tateisi et al., 2005) and rapid progress of data-driven parsers (Lease and Charniak, 2005; Nivre et al., 2007), there are now fast, robust and accurate syn- tactic parsers for text in the biomedical domain. Recent research shows that parsing accuracy of biomedical corpora is now between 80% and 90% (Clegg and Shepherd, 2007; Pyysalo et al., …

Content truncated to 500 words