Self-Training with Products of Latent Variable Grammars

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 12–22, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Self-training with Products of Latent Variable Grammars Zhongqiang Huang† †UMIACS University of Maryland College Park, MD zqhuang@umd.edu Mary Harper†‡ ‡HLT Center of Excellence Johns Hopkins University Baltimore, MD mharper@umd.edu Slav Petrov∗ ∗Google Research 76 Ninth Avenue New York, NY slav@google.com Abstract We study self-training with products of latent variable grammars in this paper. We show that increasing the quality of the automatically parsed data used for self-training gives higher accuracy self-trained grammars. Our genera- tive self-trained grammars reach F scores of 91.6 on the WSJ test set and surpass even discriminative reranking systems without self- training. Additionally, we show that multi- ple self-trained grammars can be combined in a product model to achieve even higher ac- curacy. The product model is most effective when the individual underlying grammars are most diverse. Combining multiple grammars that were self-trained on disjoint sets of un- labeled data results in a final test accuracy of 92.5% on the WSJ test set and 89.6% on our Broadcast News test set. 1 Introduction The latent variable approach of Petrov et al. (2006) is capable of learning high accuracy context-free grammars directly from a raw treebank. It starts from a coarse treebank grammar (Charniak, 1997), and uses latent variables to refine the context-free assumptions encoded in the grammar. A hierarchi- cal split-and-merge algorithm introduces grammar complexity gradually, iteratively splitting (and po- tentially merging back) each observed treebank cat- egory into a number of increasingly refined latent subcategories. The Expectation Maximization (EM) algorithm is used to train the model, guaranteeing that each EM iteration will increase the training like- lihood. However, because the latent variable gram- mars are not explicitly regularized, EM keeps fit- ting the training data and eventually begins over- fitting (Liang et al., 2007). Moreover, EM is a lo- cal method, making no promises regarding the final point of convergence when initialized from different random seeds. Recently, Petrov (2010) showed that substantial differences between the learned gram- mars remain, even if the hierarchical splitting re- duces the variance across independent runs of EM. In order to counteract the overfitting behavior, Petrov et al. (2006) introduced a linear smoothing procedure that allows training grammars for 6 split- merge (SM) rounds without overfitting. The in- creased expressiveness of the model, combined with the more robust parameter estimates provided by the smoothing, results in a nice increase in parsing ac- curacy on a held-out set. However, as reported by Petrov (2009) and Huang and Harper (2009), an ad- ditional 7th SM round actually hurts performance. Huang and Harper (2009) addressed the issue of data sparsity and overfitting from a different angle. They showed that self-training latent variable gram- mars on their own output can mitigate data spar- sity issues and improve parsing accuracy. Because the capacity of the model can grow with the size of the training data, latent variable grammars are able …

Content truncated to 500 words