Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency

Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency Dan Klein Computer Science Department Stanford University Stanford, CA 94305-9040 klein@cs.stanford.edu Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305-9040 manning@cs.stanford.edu Abstract We present a generative model for the unsupervised learning of dependency structures. We also describe the multiplicative combination of this dependency model with a model of linear constituency. The product model outperforms both components on their respective evalu- ation metrics, giving the best published figures for un- supervised dependency parsing and unsupervised con- stituency parsing. We also demonstrate that the com- bined model works and is robust cross-linguistically, be- ing able to exploit either attachment or distributional reg- ularities that are salient in the data. 1 Introduction The task of statistically inducing hierarchical syn- tactic structure over unannotated sentences of nat- ural language has received a great deal of atten- tion (Carroll and Charniak, 1992; Pereira and Sch- abes, 1992; Brill, 1993; Stolcke and Omohundro, 1994). Researchers have explored this problem for a variety of reasons: to argue empirically against the poverty of the stimulus (Clark, 2001), to use induction systems as a first stage in constructing large treebanks (van Zaanen, 2000), to build better language models (Baker, 1979; Chen, 1995), and to examine cognitive issues in language learning (Solan et al., 2003). An important distinction should be drawn between work primarily interested in the weak generative capacity of models, where model- ing hierarchical structure is only useful insofar as it leads to improved models over observed structures (Baker, 1979; Chen, 1995), and work interested in the strong generative capacity of models, where the unobserved structure itself is evaluated (van Zaa- nen, 2000; Clark, 2001; Klein and Manning, 2002). This paper falls into the latter category; we will be inducing models of linguistic constituency and de- pendency with the goal of recovering linguistically plausible structures. We make no claims as to the cognitive plausibility of the induction mechanisms we present here; however, the ability of these sys- tems to recover substantial linguistic patterns from surface yields alone does speak to the strength of support for these patterns in the data, and hence un- dermines arguments based on “the poverty of the stimulus” (Chomsky, 1965). 2 Unsupervised Dependency Parsing Most recent progress in unsupervised parsing has come from tree or phrase-structure grammar based models (Clark, 2001; Klein and Manning, 2002), but there are compelling reasons to reconsider un- supervised dependency parsing. First, most state-of- the-art supervised parsers make use of specific lexi- cal information in addition to word-class level infor- mation – perhaps lexical information could be a use- ful source of information for unsupervised methods. Second, a central motivation for using tree struc- tures in computational linguistics is to enable the extraction of dependencies – function-argument and modification structures – and it might be more ad- vantageous to induce such structures directly. Third, as we show below, for languages such as Chinese, which have few function words, and for which the definition of lexical categories is …

Content truncated to 500 words