Automatic Acquisition of Training Data for Statistical Parsers

Automatic Acquisition of Training Data for Statistical Parsers Susan Howlett and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {show0556,james}@it.usyd.edu.au Abstract The limitations of existing data sets for train- ing parsers has led to a need for additional data. However, the cost of manually annotat- ing the amount and range of data required is prohibitive. For a number of simple facts like those sought in Question Answering, we com- pile a corpus of sentences extracted from the Web that contain the fact keywords. We use a state-of-the-art parser to parse these sentences, constraining the analysis of the more complex sentences using information from the simpler sentences. This allows us to automatically cre- ate additional annotated sentences which we then use to augment our existing training data. 1 Introduction Determining the syntactic structure of sentences is a necessary step in analysing the content of text for a range of language processing tasks such as Ques- tion Answering (Harabagiu et al., 2000) and Ma- chine Translation (Melamed, 2004). The structures that a parsing system assigns to sentences are governed by the grammar used. While some parsers make use of hand-crafted grammars, e.g. Riezler et al. (2002) and Briscoe et al. (2006), these typically cannot accommodate a wide variety of sentences and increasing coverage incurs signif- icant development costs (Cahill et al., 2008). This has led to interest in automatic acquisition of gram- mars from raw text or automatically annotated data such as the Penn Treebank (Marcus et al., 1993). Automatically-acquired grammars may be clas- sified according to whether the learning algorithm used to estimate the language model is supervised or unsupervised. Supervised algorithms, e.g. Collins (1999) and Charniak (2000), require a large num- ber of sentences already parsed according to the desired formalism, while unsupervised approaches, e.g. Bod (2006) and Seginer (2007), operate directly on raw text. While supervised approaches have gen- erally proven more successful, the need for anno- tated training data is a major bottleneck. Although the emergence of the Penn Treebank as a standard resource has been beneficial in parser development and evaluation, parsing performance drops when analysing text from domains other than that represented in the training data (Sekine, 1997; Gildea, 2001). In addition, there is evidence that lan- guage processing performance can still benefit from orders of magnitude more data (e.g. Banko and Brill (2001)). However, the cost of manually annotating the necessary amount of data is prohibitive. We investigate a method of automatically creat- ing annotated data to supplement existing training corpora. We constructed a list of facts based on fac- toid questions from the TREC 2004 Question An- swering track (Voorhees, 2004) and the ISI Question Answer Typology (Hovy et al., 2002). For each of these facts, we extracted sentences from Web text that contained all the keywords of the fact. These sentences were then parsed using a state-of-the-art parser (Clark and Curran, 2007). By assuming that the same grammatical relations always hold between the keywords, we use the anal- …

Content truncated to 500 words