Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 74–82, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction Shay B. Cohen and Noah A. Smith Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA {scohen,nasmith}@cs.cmu.edu Abstract We present a family of priors over probabilis- tic grammar weights, called the shared logistic normal distribution. This family extends the partitioned logistic normal distribution, en- abling factored covariance between the prob- abilities of different derivation events in the probabilistic grammar, providing a new way to encode prior knowledge about an unknown grammar. We describe a variational EM al- gorithm for learning a probabilistic grammar based on this family of priors. We then experi- ment with unsupervised dependency grammar induction and show significant improvements using our model for both monolingual learn- ing and bilingual learning with a non-parallel, multilingual corpus. 1 Introduction Probabilistic grammars have become an important tool in natural language processing. They are most commonly used for parsing and linguistic analy- sis (Charniak and Johnson, 2005; Collins, 2003), but are now commonly seen in applications like ma- chine translation (Wu, 1997) and question answer- ing (Wang et al., 2007). An attractive property of probabilistic grammars is that they permit the use of well-understood parameter estimation methods for learning—both from labeled and unlabeled data. Here we tackle the unsupervised grammar learning problem, specifically for unlexicalized context-free dependency grammars, using an empirical Bayesian approach with a novel family of priors. There has been an increased interest recently in employing Bayesian modeling for probabilistic grammars in different settings, ranging from putting priors over grammar probabilities (Johnson et al., 2007) to putting non-parametric priors over deriva- tions (Johnson et al., 2006) to learning the set of states in a grammar (Finkel et al., 2007; Liang et al., 2007). Bayesian methods offer an elegant frame- work for combining prior knowledge with data. The main challenge in Bayesian grammar learning is efficiently approximating probabilistic inference, which is generally intractable. Most commonly vari- ational (Johnson, 2007; Kurihara and Sato, 2006) or sampling techniques are applied (Johnson et al., 2006). Because probabilistic grammars are built out of multinomial distributions, the Dirichlet family (or, more precisely, a collection of Dirichlets) is a natural candidate for probabilistic grammars because of its conjugacy to the multinomial family. Conjugacy im- plies a clean form for the posterior distribution over grammar probabilities (given the data and the prior), bestowing computational tractability. Following work by Blei and Lafferty (2006) for topic models, Cohen et al. (2008) proposed an alter- native to Dirichlet priors for probabilistic grammars, based on the logistic normal (LN) distribution over the probability simplex. Cohen et al. used this prior to softly tie grammar weights through the covariance parameters of the LN. The prior encodes informa- tion about which grammar rules’ weights are likely to covary, a more intuitive and expressive represen- tation of knowledge than offered by …

Content truncated to 500 words