Better Informed Training of Latent Syntactic Features

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 317–326, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Better Informed Training of Latent Syntactic Features Markus Dreyer and Jason Eisner Department of Computer Science / Center for Language and Speech Processing Johns Hopkins University 3400 North Charles Street, Baltimore, MD 21218 USA {markus,jason}@clsp.jhu.edu Abstract We study unsupervised methods for learn- ing refinements of the nonterminals in a treebank. Following Matsuzaki et al. (2005) and Prescher (2005), we may for example split NP without supervision into NP[0] and NP[1], which behave differently. We first propose to learn a PCFG that adds such features to nonterminals in such a way that they respect patterns of linguis- tic feature passing: each node’s nontermi- nal features are either identical to, or inde- pendent of, those of its parent. This lin- guistic constraint reduces runtime and the number of parameters to be learned. How- ever, it did not yield improvements when training on the Penn Treebank. An orthog- onal strategy was more successful: to im- prove the performance of the EM learner by treebank preprocessing and by anneal- ing methods that split nonterminals selec- tively. Using these methods, we can main- tain high parsing accuracy while dramati- cally reducing the model size. 1 Introduction Treebanks never contain enough information; thus PCFGs estimated straightforwardly from the Penn Treebank (Bies et al., 1995) work only moderately well (Charniak, 1996). To address this problem, researchers have used heuristics to add more infor- mation. Eisner (1996), Charniak (1997), Collins (1997), and many subsequent researchers1 anno- tated every node with lexical features passed up from its “head child,” in order to more precisely re- flect the node’s “inside” contents. Charniak (1997) and Johnson (1998) annotated each node with its parent and grandparent nonterminals, to more pre- cisely reflect its “outside” context. Collins (1996) split the sentence label S into two versions, repre- senting sentences with and without subjects. He 1Not to mention earlier non-PCFG lexicalized statistical parsers, notably Magerman (1995) for the Penn Treebank. also modified the treebank to contain different la- bels for standard and for base noun phrases. Klein and Manning (2003) identified nonterminals that could valuably be split into fine-grained ones us- ing hand-written linguistic rules. Their unlexical- ized parser combined several such heuristics with rule markovization and reached a performance similar to early lexicalized parsers. In all these cases, choosing which nonterminals to split, and how, was a matter of art. Ideally such splits would be learned automatically from the given treebank itself. This would be less costly and more portable to treebanks for new domains and languages. One might also hope that the auto- matically learned splits would be more effective. Matsuzaki et al. (2005) introduced a model for such learning: PCFG-LA.2 They used EM to in- duce fine-grained versions of a given treebank’s nonterminals and rules. We present models that similarly learn to propagate fine-grained features through the tree, but only in certain linguistically motivated ways. Our models …

Content truncated to 500 words