Feature-Rich Log-Linear Lexical Model for Latent Variable PCFG Grammars

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 219–227, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Feature-Rich Log-Linear Lexical Model for Latent Variable PCFG Grammars Zhongqiang Huang and Mary Harper Department of Computer Science University of Maryland, College Park {zqhuang,mharper}@umd.edu Abstract Context-free grammars with latent anno- tations (PCFG-LA) have been found to be effective for parsing many languages; however, currently their lexical model may be subject to over-fitting and requires language engineering to handle out-of- vocabulary (OOV) words. Inspired by pre- vious studies that have incorporated rich features into generative models, we pro- pose to use a feature-rich log-linear lexical model to train PCFG-LA grammars that are more robust to rare and OOV words. The proposed lexical model has three ad- vantages: over-fitting is alleviated via reg- ularization, OOV words are modeled us- ing rich features, and lexical features are exploited for grammar induction. Our ap- proach results in significantly more accu- rate PCFG-LA grammars that are flexible to train for different languages (with test F scores of 90.5, 85.0, and 81.9 on WSJ, CTB6, and ATB, respectively). 1 Introduction The latent variable approach of (Matsuzaki et al., 2005; Petrov et al., 2006) is capable of learning high accuracy context-free grammars di- rectly from a raw treebank, and has achieved state-of-the-art parsing accuracies on multiple lan- guages, outperforming many other parsers that are engineered for performance in a particular lan- guage (Petrov, 2009; Green and Manning, 2010). However, the lexical model of PCFG-LA gram- mars (responsible for emitting words from latent POS tags) is not designed to effectively handle OOV words universally. In fact, hand-crafted rules designed for English OOV words were used in the multi-language study of (Petrov, 2009) for non- English languages, leaving room for further im- provement for each of the languages studied. Huang and Harper (2009) and Attia et al. (2010) studied the impact of rare and OOV word handling for parsing with PCFG-LA grammars, especially for non-English languages. They both found that language-specific handling of OOV words signif- icantly improves parsing performance. However, hand tailoring of the language-specific module with expert knowledge may produce suboptimal results, and would not be applicable to new lan- guages. Petrov and Klein (2008) presented a dis- criminatively trained PCFG-LA model that makes use of rich morphological features for handling OOV words and obtained improved performance on some languages; however, this method was considerably less accurate than its strong gener- ative counterpart on English WSJ. Berg-Kirkpatrick et al. (2010) demonstrated that each generation step of a generative process can be modeled as a locally normalized log-linear model so that rich features can be incorporated for learning unsupervised models, e.g., POS in- duction. Inspired by their work, we propose a log-linear lexical model for generative PCFG-LA grammars. It maintains the advantages of gener- ative models, while providing a principled way to: 1) alleviate over-fitting via regularization, 2) handle OOV words using rich features, and 3) ex- ploit lexical features for grammar induction. The proposed …

Content truncated to 500 words