Representational Bias in Unsupervised Learning of Syllable Structure

Proceedings of the 9th Conference on Computational Natural Language Learning (CoNLL), pages 112–119, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Representational Bias in Unsupervised Learning of Syllable Structure Sharon Goldwater and Mark Johnson Department of Cognitive and Linguistic Sciences Brown University Providence, RI 02912 {Sharon Goldwater,Mark Johnson}@brown.edu Abstract Unsupervised learning algorithms based on Expectation Maximization (EM) are often straightforward to implement and provably converge on a local likelihood maximum. However, these algorithms of- ten do not perform well in practice. Com- mon wisdom holds that they yield poor results because they are overly sensitive to initial parameter values and easily get stuck in local (but not global) maxima. We present a series of experiments indi- cating that for the task of learning sylla- ble structure, the initial parameter weights are not crucial. Rather, it is the choice of model class itself that makes the differ- ence between successful and unsuccess- ful learning. We use a language-universal rule-based algorithm to find a good set of parameters, and then train the parameter weights using EM. We achieve word ac- curacy of 95.9% on German and 97.1% on English, as compared to 97.4% and 98.1% respectively for supervised training. 1 Introduction The use of statistical methods in computational lin- guistics has produced advances in tasks such as pars- ing, information retrieval, and machine translation. However, most of the successful work to date has used supervised learning techniques. Unsupervised algorithms that can learn from raw linguistic data, as humans can, remain a challenge. In a statistical framework, one method that can be used for unsu- pervised learning is to devise a probabilistic model of the data, and then choose the values for the model parameters that maximize the likelihood of the data under the model. If the model contains hidden variables, there is often no closed-form expression for the maximum likelihood parameter values, and some iterative ap- proximation method must be used. Expectation Maximization (EM) (Neal and Hinton, 1998) is one way to find parameter values that at least lo- cally maximize the likelihood for models with hid- den variables. EM is attractive because at each iteration, the likelihood of the data is guaranteed not to decrease. In addition, there are efficient dynamic-programming versions of the EM algo- rithm for several classes of models that are important in computational linguistics, such as the forward- backward algorithm for training Hidden Markov Models (HMMs) and the inside-outside algorithm for training Probabilistic Context-Free Grammars (PCFGs). Despite the advantages of maximum likelihood estimation and its implementation via various in- stantiations of the EM algorithm, it is widely re- garded as ineffective for unsupervised language learning. Merialdo (1994) showed that with only a tiny amount of tagged training data, supervised training of an HMM part-of-speech tagger outper- formed unsupervised EM training. Later results (e.g. Brill (1995)) seemed to indicate that other methods of unsupervised learning could be more effective (al- though the work of Banko and Moore (2004) sug- gests that the difference may be far less …

Content truncated to 500 words