Active learning for HPSG parse selection

Active learning for HPSG parse selection Jason Baldridge and Miles Osborne School of Informatics University of Edinburgh Edinburgh EH8 9LW, UK � jmb,osborne � @cogsci.ed.ac.uk Abstract We describe new features and algorithms for HPSG parse selection models and address the task of creating annotated material to train them. We evaluate the ability of several sam- ple selection methods to reduce the number of annotated sentences necessary to achieve a given level of performance. Our best method achieves a 60% reduction in the amount of training material without any loss in accuracy. 1 Introduction Even with significant resources such as the Penn Tree- bank, a major bottleneck for improving statistical parsers has been the lack of sufficient annotated material from which to estimate their parameters. Most statistical pars- ing research, such as Collins (1997), has centered on training probabilistic context-free grammars using the Penn Treebank. For richer linguistic frameworks, such as Head-Driven Phrase Structure Grammar (HPSG), there is even less annotated material available for training stochastic parsing models. There is thus a pressing need to create significant volumes of annotated material in a logistically efficient manner. Even if it were possible to bootstrap from the Penn Treebank, it is still unlikely that there would be sufficient quantities of high quality mate- rial. There has been a strong focus in recent years on us- ing the active learning technique of selective sampling to reduce the amount of human-annotated training material needed to train models for various natural language pro- cessing tasks. The aim of selective sampling is to iden- tify the most informative examples, according to some se- lection method, from a large pool of unlabelled material. Such selected examples are then manually labelled. Se- lective sampling has typically been applied to classifica- tion tasks, but has also been shown to reduce the number of examples needed for inducing Lexicalized Tree Inser- tion Grammars from the Penn Treebank (Hwa, 2000). The suitability of active learning for HPSG-type gram- mars has as yet not been explored. This paper addresses the problem of minimizing the human effort expended in creating annotated training material for HPSG parse se- lection by using selective sampling. We do so in the con- text of Redwoods (Oepen et al., 2002), a treebank that contains HPSG analyses for sentences from the Verbmo- bil appointment scheduling and travel planning domains. We show that sample selection metrics based on tree en- tropy (Hwa, 2000) and disagreement between two differ- ent parse selection models significantly reduce the num- ber of annotated sentences necessary to match a given level of performance according to random selection. Fur- thermore, by combining these two methods as an ensem- ble selection method, we require even fewer examples — achieving a 60% reduction in the amount of annotated training material needed to outperform a model trained on randomly selected material. These results suggest that significant reductions in human effort can be real- ized through selective sampling when creating annotated material for linguistically rich grammar formalisms. As the basis of …

Content truncated to 500 words