The Return of Lexical Dependencies: Neural Lexicalized PCFGs

The Return of Lexical Dependencies: Neural Lexicalized PCFGs Hao Zhu, Yonatan Bisk, Graham Neubig Language Technologies Institute, Carnegie Mellon University zhuhao@cmu.edu, {ybisk, gneubig}@cs.cmu.edu Abstract In this paper we demonstrate that context free grammar (CFG) based methods for grammar induction benefit from modeling lexical depen- dencies. This contrasts to the most popular current methods for grammar induction, which focus on discovering either constituents or dependencies. Previous approaches to marry these two disparate syntactic formalisms (e.g., lexicalized PCFGs) have been plagued by sparsity, making them unsuitable for unsu- pervised grammar induction. However, in this work, we present novel neural models of lex- icalized PCFGs that allow us to overcome sparsity problems and effectively induce both constituents and dependencies within a sin- gle model. Experiments demonstrate that this unified framework results in stronger results on both representations than achieved when modeling either formalism alone.1 1 Introduction Unsupervised grammar induction aims at building a formal device for discovering syntactic structure from natural language corpora. Within the scope of grammar induction, there are two main direc- tions of research: unsupervised constituency pars- ing, which attempts to discover the underlying structure of phrases, and unsupervised depen- dency parsing, which attempts to discover the underlying relations between words. Early work on induction of syntactic structure focused on learning phrase structure and generally used some variant of probabilistic context-free grammars (PCFGs; Lari and Young, 1990; Charniak, 1996; Clark, 2001). In recent years, dependency grammars have gained favor as an alternative 1Code is available at https://github.com /neulab/neural-lpcfg. syntactic formulation (Yuret, 1998; Carroll and Charniak, 1992; Paskin, 2002). Specifically,thede- pendency model with valence (DMV) (Klein and Manning, 2004) forms the basis for many mod- ern approaches in dependency induction. Most recent models for grammar induction, be they for PCFGs, DMVs, or other formulations, have gen- erally coupled these models with some variety of neural model to use embeddings to capture word similarities, improve the flexibility of model para- meterization, or both (He et al., 2018; Jin et al., 2019; Kim et al., 2019; Han et al., 2019). Notably, the two different syntactic formalisms capture very different views of syntax. Phrase structure takes advantage of an abstracted recur- sive view of language, while the dependencystruc- ture more concretely focuses on the propensity of particular words in a sentence to relate to each other syntactically. However, few attempts at un- supervised grammar induction have been made to marry the two and let both benefit each other. This is precisely the issue we attempt to tackle in this paper. As a specific formalism that allows us to model both formalisms at once, we turn to lexicalized probabilistic context-free grammars (L-PCFGs; Collins, 2003). L-PCFGs borrow the underlying machinery from PCFGs but expand the grammar by allowing rules to include information about the lexical heads of each phrase, an example of which is shown in Figure 1. The head annotation in the L-PCFG provides lexical dependencies that can be informative in estimating the probabilities of generation rules. For example, the probability of VP[CHASING] → VBZ[IS] …

Content truncated to 500 words