A Semi-Automatic, Iterative Method for Creating a Domain-Specific Treebank

Proceedings of Recent Advances in Natural Language Processing, pages 413–419, Hissar, Bulgaria, 12-14 September 2011. A Semi-Automatic, Iterative Method for Creating a Domain-Specific Treebank Corina Dima Erhard Hinrichs Department of Linguistics University of T¨ubingen corina.dima, erhard.hinrichs@uni-tuebingen.de Abstract In this paper we present the development process of NLP-QT, a question treebank that will be used for data-driven parsing in the context of a domain-specific QA sys- tem for querying NLP resource metadata. We motivate the need to build NLP-QT as a resource in its own right, by com- paring the Penn Treebank-style annotation scheme used for QuestionBank (Judge et al., 2006) with the modified NP annota- tion for the Penn Treebank introduced by Vadas and Curran (2007). We argue that this modified annotation scheme provides a better interface representation for seman- tic interpretation and show how it can be incorporated into the NLP-QT resource, without significant loss in parser perfor- mance. The parsing experiments reported in the paper confirm the feasibility of an iter- ative, semi-automatic construction of the NLP-QT resource similar to the approach taken for QuestionBank. At the same time, we propose to improve the iterative re- finement technique used for QuestionBank by adopting Hwa (2001)’s heuristics for selecting additional material to be hand- corrected and added to the data set at each iteration. 1 Introduction Question-Answering (QA) systems have a long history in the field of natural language process- ing. In the 1970s and 1980s QA systems fo- cused on natural language interfaces to domain- specific data bases or expert systems. Such sys- tems typically used hand-crafted, rule-based front ends for parsing and semantic interpretation. With the increased availability of large-scale textual re- sources, QA systems more recently have focused on domain-independent broad-coverage informa- tion retrieval applications that typically employ more shallow processing techniques for question analysis and answer matching. The intended application for the research re- ported in the present paper is more in the tradi- tion of the earlier, domain-specific QA systems in that it aims to provide a natural language front-end to large repositories of metadata about language tools and resources that are made available by the CLARIN1 project. However, instead of relying on a parser with hand-crafted grammar rules, it em- ploys a robust data-driven parser that requires an- notated training data in the form of a treebank. Since the natural language front end for the in- tended QA system is English, the simplest solu- tion would be to use a statistical parser such as the Berkeley (Petrov and Klein, 2007) or Stanford (Klein and Manning, 2003) parser with an existing language model obtained from the Penn Treebank (Marcus et al., 1993). However, it is well known that parser performance drops when analyzing text from domains other than that represented in the training data (Sekine, 1997; Gildea, 2001). In par- ticular, Judge et al. (2006) have shown that lan- guage models obtained from the Penn Treebank perform far worse on questions than on their orig- inal test data. The Bikel (2004) parser they employ …

Content truncated to 500 words