Grammar Induction from Text Using Small Syntactic Prototypes

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 438–446, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Grammar Induction from Text Using Small Syntactic Prototypes Prachya Boonkwan†,‡ p.boonkwan@sms.ed.ac.uk prachya.boonkwan@nectec.or.th † School of Informatics University of Edinburgh 10 Crichton Street Edinburgh EH8 9AB, UK Mark Steedman† steedman@inf.ed.ac.uk ‡ National Electronics and Computer Technology Center 112 Phaholyothin Road Pathumthani 12120, Thailand Abstract We present an efficient technique to incor- porate a small number of cross-linguistic parameter settings defining default word orders to otherwise unsupervised gram- mar induction. A syntactic prototype, represented by the integrated model be- tween Categorial Grammar and depen- dency structure, generated from the lan- guage parameters, is used to prune the search space. We also propose heuristics which prefer less complex syntactic cate- gories to more complex ones in parse de- coding. The system reduces errors gen- erated by the state-of-the-art baselines for WSJ10 (1% error reduction of F1 score for the model trained on Sections 2–22 and tested on Section 23), Chinese10 (26% er- ror reduction of F1), German10 (9% er- ror reduction of F1), and Japanese10 (8% error reduction of F1), and is not sig- nificantly different from the baseline for Czech10. 1 Introduction Unsupervised grammar induction has gained gen- eral interest for several decades, offering the pos- sibility of building practical syntactic parsers by reducing the labor of constructing a treebank from scratch. One approach is to exploit the In- side/Outside Algorithm (Baker, 1979; Carroll and Charniak, 1992), a variation of EM algorithm for PCFG, to estimate the parameters of the parser’s language models. More recent advances in this ap- proach are the constituent-context model (CCM) (Klein and Manning, 2001; Klein and Manning, 2002), dependency model with valence (DMV) based on Collin’s head dependency model (1999), and the CCM+DMV mixture (Klein and Manning, 2004; Klein, 2005). Several search techniques and models have been added to CCM+DMV for deal- ing with local optima and data sparsity (Smith, 2006; Cohen et al., 2008; Headden III et al., 2009). Spitkovsky et al. (2010) proposed a training strat- egy where the model fully trained on shorter sen- tences and roughly trained on longer sentences tend to outperform the model fully trained on the entire dataset. Recently, Gillenwater et al. (2010) proposed the use of posterior regularization in EM in which the posterior distribution of parent-child POS tags are regulated to an expected distribution. However, purely unsupervised learning still does not perform well because the parameter es- timation can be misled by unexpected frequent cooccurrence. A common example of it is the col- location of a verb (VBZ) and a determiner (DT) in a verb phrase. This collocation results in incorrect trees such as ((VBZ DT) NN). To avoid this problem, the use of syntactic pro- totypes has been proposed. Instead of enumerat- ing every possibility, syntactic structures are cau- tiously constructed regarding some syntactic con- straints. Haghighi and Klein (2006) proposed the use of bracketing rules extracted from WSJ10 in CCM and considerably improved …

Content truncated to 500 words