Active Learning for Statistical Natural Language Parsing

Active Learning for Statistical Natural Language Parsing Min Tang Spoken Language Systems Group MIT Laboratory for Computer Science Cambridge, Massachusetts 02139, USA � mtang@sls.lcs.mit.edu � Xiaoqiang Luo Salim Roukos IBM T.J. Watson Research Center Yorktown Heights, NY 10598 � xiaoluo,roukos@us.ibm.com � Abstract It is necessary to have a (large) annotated cor- pus to build a statistical parser. Acquisition of such a corpus is costly and time-consuming. This paper presents a method to reduce this demand using active learning, which selects what samples to annotate, instead of annotating blindly the whole training corpus. Sample selection for annotation is based upon “representativeness” and “usefulness”. model-based distance is proposed to measureA the difference of two sentences and their most likely parse trees. Based on this distance, the active learning process analyzes the sample dis- tribution by clustering and calculates the den- sity of each sample to quantify its representa- tiveness. Further more, a sentence is deemed as useful if the existing model is highly uncertain about its parses, where uncertainty is measured by various entropy-based scores. Experiments are carried out in the shallow se- mantic parser of an air travel dialog system. Our result shows that for about the same pars- ing accuracy, we only need to annotate a third of the samples as compared to the usual random selection method. 1 Introduction A prerequisite for building statistical parsers (Jelinek et al., 1994; Collins, 1996; Ratnaparkhi, 1997; Charniak, 1997) is the availability of a (large) corpus of parsed sen- tences. Acquiring such a corpus is expensive and time- consuming and is often the bottleneck to build a parser for a new application or domain. The goal of this study is to reduce the amount of annotated sentences (and hence the development time) required for a statistical parser to achieve a satisfactory performance using active learning. Active learning has been studied in the context of many natural language processing (NLP) applications such as information extraction(Thompson et al., 1999), text clas- sification(McCallum and Nigam, 1998) and natural lan- guage parsing(Thompson et al., 1999; Hwa, 2000), to name a few. The basic idea is to couple tightly knowl- edge acquisition, e.g., annotating sentences for parsing, with model-training, as opposed to treating them sepa- rately. In our setup, we assume that a small amount of annotated sentences is initially available, which is used to build a statistical parser. We also assume that there is a large corpus of unannotated sentences at our disposal – this corpus is called active training set. A batch of sam- ples1 is selected using algorithms developed here, and are annotated by human beings and are then added to training data to rebuild the model. The procedure is iterated until the model reaches a certain accuracy level. Our efforts are devoted to two aspects: first, we be- lieve that the selected samples should reflect the underly- ing distribution of the training corpus. In other words, the selected samples need to be representative. To this end, a model-based structural distance is defined to quantify …

Content truncated to 500 words