Cross-Task Knowledge-Constrained Self Training

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 680–688, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Cross-Task Knowledge-Constrained Self Training Hal Daum´e III School of Computing University of Utah Salt Lake City, UT 84112 me@hal3.name Abstract We present an algorithmic framework for learning multiple related tasks. Our frame- work exploits a form of prior knowledge that relates the output spaces of these tasks. We present PAC learning results that analyze the conditions under which such learning is pos- sible. We present results on learning a shal- low parser and named-entity recognition sys- tem that exploits our framework, showing con- sistent improvements over baseline methods. 1 Introduction When two NLP systems are run on the same data, we expect certain constraints to hold between their out- puts. This is a form of prior knowledge. We propose a self-training framework that uses such information to significantly boost the performance of one of the systems. The key idea is to perform self-training only on outputs that obey the constraints. Our motivating example in this paper is the task pair: named entity recognition (NER) and shallow parsing (aka syntactic chunking). Consider a hid- den sentence with known POS and syntactic struc- ture below. Further consider four potential NER se- quences for this sentence. POS: NNP NNP VBD TO NNP NN Chunk: [- NP -][-VP-][-PP-][-NP-][-NP-] NER1: [- Per -][- O -][-Org-][- 0 -] NER2: [- Per -][- O -][- O -][- O -][- O -] NER3: [- Per -][- O -][- O -][- Org -] NER4: [- Per -][- O -][- O -][-Org-][- O -] Without ever seeing the actual sentence, can we guess which NER sequence is correct? NER1 seems wrong because we feel like named entities should not be part of verb phrases. NER2 seems wrong be- cause there is an NNP1 (proper noun) that is not part of a named entity (word 5). NER3 is amiss because we feel it is unlikely that a single name should span more than one NP (last two words). NER4 has none of these problems and seems quite reasonable. In fact, for the hidden sentence, NER4 is correct2. The remainder of this paper deals with the prob- lem of formulating such prior knowledge into a workable system. There are similarities between our proposed model and both self-training and co- training; background is given in Section 2. We present a formal model for our approach and per- form a simple, yet informative, analysis (Section 3). This analysis allows us to define what good and bad constraints are. Throughout, we use a running example of NER using hidden Markov models to show the efficacy of the method and the relation- ship between the theory and the implementation. Fi- nally, we present full-blown results on seven dif- ferent NER data sets (one from CoNLL, six from ACE), comparing our method to several competi- tive baselines (Section 4). We see that for many of these data sets, less than one hundred labeled NER sentences …

Content truncated to 500 words