An HDP Model for Inducing Combinatory Categorial Grammars

Transactions of the Association for Computational Linguistics, 1 (2013) 75–88. Action Editor: Sharon Goldwater. Submitted 11/2012; Revised 1/2013; Published 3/2013. c⃝2013 Association for Computational Linguistics. An HDP Model for Inducing Combinatory Categorial Grammars Yonatan Bisk and Julia Hockenmaier Department of Computer Science The University of Illinois at Urbana-Champaign 201 N Goodwin Ave Urbana, IL 61801 {bisk1,juliahmr}@illinois.edu Abstract We introduce a novel nonparametric Bayesian model for the induction of Combinatory Cat- egorial Grammars from POS-tagged text. It achieves state of the art performance on a number of languages, and induces linguisti- cally plausible lexicons. 1 Introduction What grammatical representation is appropriate for unsupervised grammar induction? Initial attempts with context-free grammars (CFGs) were not very successful (Carroll and Charniak, 1992; Charniak, 1993). One reason may be that CFGs require the specification of a finite inventory of nonterminal cat- egories and rewrite rules, but unless one adopts lin- guistic principles such as X-bar theory (Jackendoff, 1977), these nonterminals are essentially arbitrary labels that can be combined in arbitrary ways. While further CFG-based approaches have been proposed (Clark, 2001; Kurihara and Sato, 2004), most re- cent work has followed Klein and Manning (2004) in developing models for the induction of projec- tive dependency grammars. It has been shown that more sophisticated probability models (Headden III et al., 2009; Gillenwater et al., 2011; Cohen and Smith, 2010) and learning regimes (Spitkovsky et al., 2010), as well as the incorporation of prior lin- guistic knowledge (Cohen and Smith, 2009; Berg- Kirkpatrick and Klein, 2010; Naseem et al., 2010) can lead to significant improvement over Klein and Manning’s baseline model. The use of dependency grammars circumvents the question of how to obtain an appropriate inventory of categories, since depen- dency parses are simply defined by unlabeled edges between the lexical items in the sentence. But de- pendency grammars make it also difficult to cap- ture non-local structures, and Blunsom and Cohn (2010) show that it may be advantageous to refor- mulate the underlying dependency grammar in terms of a tree-substitution grammar (TSG) which pairs words with treelets that specify the number of left and right dependents they have. In this paper, we explore yet another option: instead of dependency grammars, we use Combinatory Categorial Gram- mar (CCG, Steedman (1996; 2000)), a linguistically expressive formalism that pairs lexical items with rich categories that capture all language-specific in- formation. This may seem a puzzling choice, since CCG requires a significantly larger inventory of cat- egories than is commonly assumed for CFGs. How- ever, unlike CFG nonterminals, CCG categories are not arbitrary symbols: they encode, and are deter- mined by, the basic word order of the language and the number of arguments each word takes. CCG is very similar to TSG in that it also pairs lexical items with rich items that capture all language-specific in- formation. Like TSG and projective dependency grammars, we restrict ourselves to a weakly context- free fragment of CCG. But while TSG does not dis- tinguish between argument and modifier dependen- cies, CCG makes an …

Content truncated to 500 words