Improving nonparameteric Bayesian inference: experiments on unsupervised word segmentation with adaptor grammars

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 317–325, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Improving nonparameteric Bayesian inference: experiments on unsupervised word segmentation with adaptor grammars Mark Johnson Brown University Providence, RI Mark Johnson@Brown.edu Sharon Goldwater University of Edinburgh Edinburgh EH8 9AB sgwater@inf.ed.ac.uk Abstract One of the reasons nonparametric Bayesian inference is attracting attention in computa- tional linguistics is because it provides a prin- cipled way of learning the units of generaliza- tion together with their probabilities. Adaptor grammars are a framework for defining a va- riety of hierarchical nonparametric Bayesian models. This paper investigates some of the choices that arise in formulating adap- tor grammars and associated inference proce- dures, and shows that they can have a dra- matic impact on performance in an unsuper- vised word segmentation task. With appro- priate adaptor grammars and inference proce- dures we achieve an 87% word token f-score on the standard Brent version of the Bernstein- Ratner corpus, which is an error reduction of over 35% over the best previously reported re- sults for this corpus. 1 Introduction Most machine learning algorithms used in computa- tional linguistics are parametric, i.e., they learn a nu- merical weight (e.g., a probability) associated with each feature, where the set of features is fixed be- fore learning begins. Such procedures can be used to learn features or structural units by embedding them in a “propose-and-prune” algorithm: a feature proposal component proposes potentially useful fea- tures (e.g., combinations of the currently most useful features), which are then fed to a parametric learner that estimates their weights. After estimating fea- ture weights and pruning “useless” low-weight fea- tures, the cycle repeats. While such algorithms can achieve impressive results (Stolcke and Omohundro, 1994), their effectiveness depends on how well the feature proposal step relates to the overall learning objective, and it can take considerable insight and experimentation to devise good feature proposals. One of the main reasons for the recent interest in nonparametric Bayesian inference is that it offers a systematic framework for structural inference, i.e., inferring the features relevant to a particular prob- lem as well as their weights. (Here “nonparamet- ric” means that the models do not have a fixed set of parameters; our nonparametric models do have pa- rameters, but the particular parameters in a model are learned along with their values). Dirichlet Pro- cesses and their associated predictive distributions, Chinese Restaurant Processes, are one kind of non- parametric Bayesian model that has received consid- erable attention recently, in part because they can be composed in hierarchical fashion to form Hierarchi- cal Dirichlet Processes (HDP) (Teh et al., 2006). Lexical acquisition is an ideal test-bed for explor- ing methods for inferring structure, where the fea- tures learned are the words of the language. (Even the most hard-core nativists agree that the words of a language must be learned). We use the unsupervised word segmentation problem as a test case for eval- uating structural inference …

Content truncated to 500 words