Long-Tail Distributions and Unsupervised Learning of Morphology

Proceedings of COLING 2012: Technical Papers, pages 3121–3136, COLING 2012, Mumbai, December 2012. Long-tail Distributions and Unsupervised Learning of Morphology Qiuye Zhao1 Mitch Marcus1 (1) University Of Pennsylvania qiuye@cis.upenn.edu, mitch@cis.upenn.edu Abstract In previous work on unsupervised learning of morphology, the long-tail pattern in the rank-frequency distribution of words, as well as of morphological units, is usually considered as following Zipf’s law (power-law). We argue that these long-tail distributions can also be considered as lognormal. Since we know the conjugate prior distribution for a lognormal likelihood, we propose to generate morphology data from lognormal distributions. When the performance is evaluated by a token- based criterion, giving more weights to the results of frequent words, the proposed model preforms significantly better than other models in discussion. Moreover, we capture the statistical properties of morphological units with a Bayesian approach, other than a rule-based approach as studied in (Chan, 2008) and (Zhao and Marcus, 2011). Given the multiplicative property of lognormal distributions, we can directly capture the long-tail distribution of word frequency, without the need of an additional generative process as studied in (Goldwater et al., 2006). Keywords: Morphological Learning, Zipf’s law, Lognormal distribution, Long tail distribution, Gibbs Sampling, Bayesian approach. 3121 1 Introduction Unsupervised learning of morphology is an active research area. In this work, we will focus on learning segmentations of words1. A segmentation of word w can be denoted as w = t.f , which means that concatenating stem t and suffix f gives word w. Assuming that stems and suffixes are independently distributed, a baseline generative morphology model is like this, kT, kF = number of distinct stem types, number of distinct suffix types αT,αF = constant hyper-parameters θ T,θ F ∼ Symmetric-DirichletkT (αT),Symmetric-DirichletkF (αF) ti=1...N, fi=1...N ∼ Multinomial(θ T),Multinomial(θ F) wi=1...N ∼ I(w = t.f )P(t|θ T)P(f |θ F) where N is the number of words and I(w = t.f ) is the indicator function taking on value 1 when concatenating stem t and suffix f gives word w and 0 otherwise. (a) The rank-frequency plot of words. (b) The log-log rank-frequency plot. (c) The CCDF plot of words. Figure 1: Observing long-tail distributions of word frequency in WSJ Penn Treebank. In the above baseline model, there no special statistical property of word frequency is captured, nor of morphological units (e.g. stems or suffixes). As depicted in Figure 1-a, it is well-known that given a large corpus, a rank-frequency plot of words generally exhibits a long tail that is distinctively heavier than in normal (Gaussian) distributions. An alternative way to show this long-tail pattern is by plotting the corresponding Complementary Cumulative Distribution Function (CCDF) on logistic scales. As depicted in Figure 1-c, the CCDF plot behaves like a straight line and is smoother than the corresponding log-log rank-frequency plot in Figure 1-b, which also approaches a straight line for long-tail distributions. When words are segmented into stems and suffixes, distributions of these units can also be examined. As we are going to show in Section 2.3, the rank-frequency …

Content truncated to 500 words