QuestionBank: Creating a Corpus of Parse-Annotated Questions

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 497–504, Sydney, July 2006. c⃝2006 Association for Computational Linguistics QuestionBank: Creating a Corpus of Parse-Annotated Questions John Judge1, Aoife Cahill1, and Josef van Genabith1,2 1National Centre for Language Technology and School of Computing, Dublin City University, Dublin, Ireland 2IBM Dublin Center for Advanced Studies, IBM Dublin, Ireland {jjudge,acahill,josef}@computing.dcu.ie Abstract This paper describes the development of QuestionBank, a corpus of 4000 parse- annotated questions for (i) use in training parsers employed in QA, and (ii) evalua- tion of question parsing. We present a se- ries of experiments to investigate the ef- fectiveness of QuestionBank as both an exclusive and supplementary training re- source for a state-of-the-art parser in pars- ing both question and non-question test sets. We introduce a new method for recovering empty nodes and their an- tecedents (capturing long distance depen- dencies) from parser output in CFG trees using LFG f-structure reentrancies. Our main findings are (i) using QuestionBank training data improves parser performance to 89.75% labelled bracketing f-score, an increase of almost 11% over the base- line; (ii) back-testing experiments on non- question data (Penn-II WSJ Section 23) shows that the retrained parser does not suffer a performance drop on non-question material; (iii) ablation experiments show that the size of training material provided by QuestionBank is sufficient to achieve optimal results; (iv) our method for recov- ering empty nodes captures long distance dependencies in questions from the ATIS corpus with high precision (96.82%) and low recall (39.38%). In summary, Ques- tionBank provides a useful new resource in parser-based QA research. 1 Introduction Parse-annotated corpora (treebanks) are crucial for developing machine learning and statistics-based parsing resources for a given language or task. Large treebanks are available for major languages, however these are often based on a specific text type or genre, e.g. financial newspaper text (the Penn-II Treebank (Marcus et al., 1993)). This can limit the applicability of grammatical resources in- duced from treebanks in that such resources un- derperform when used on a different type of text or for a specific task. In this paper we present work on creating Ques- tionBank, a treebank of parse-annotated questions, which can be used as a supplementary training re- source to allow parsers to accurately parse ques- tions (as well as other text). Alternatively, the re- source can be used as a stand-alone training corpus to train a parser specifically for questions. Either scenario will be useful in training parsers for use in question answering (QA) tasks, and it also pro- vides a suitable resource to evaluate the accuracy of these parsers on questions. We use a semi-automatic “bootstrapping” method to create the question treebank from raw text. We show that a parser trained on the ques- tion treebank alone can accurately parse ques- tions. Training on a combined corpus consisting of the question treebank and an established training set (Sections 02-21 of the Penn-II Treebank), the parser gives state-of-the-art performance on both questions …

Content truncated to 500 words