Inducing Syntactic Categories by Context Distribution Clustering

In: Proceedings of CoNLL-2000 and LLL-2000, pages 91-94, Lisbon, Portugal, 2000. Inducing Syntactic Categories by Context Distribution Clustering Alexander Clark School of Cognitive and Computing Sciences University of Sussex alexc¢cogs, susx. ac. uk Abstract This paper addresses the issue of the automatic induction of syntactic categories from unanno- tared corpora. Previous techniques give good results, but fail to cope well with ambiguity or rare words. An algorithm, context distribution clustering (CDC), is presented which can be naturally extended to handle these problems. 1 Introduction In this paper I present a novel program that in- duces syntactic categories from comparatively small corpora of unlabelled text, using only dis- tributional information. There are various mo- tivations for this task, which affect the algo- rithms employed. Many NLP systems use a set of tags, largely syntactic in motivation, that have been selected according to various criteria. In many circumstances it would be desirable for engineering reasons to generate a larger set of tags, or a set of domain-specific tags for a par- ticular corpus. Furthermore, the construction of cognitive models of language acquisition- that will almost certainly involve some notion of syntactic category - requires an explanation of the acquisition of that set of syntactic cate- gories. The amount of data used in this study is 12 million words, which is consistent with a pessimistic lower bound on the linguistic experi- ence of the infant language learner in the period from 2 to 5 years of age, and has had capitalisa- tion removed as being information not available in that circumstance. 2 Previous Work Previous work falls into two categories. A num- ber of researchers have obtained good results using pattern recognition techniques. Finch and Chater (1992), (1995) and Schfitze (1993), (1997) use a set of features derived from the co-occurrence statistics of common words to- gether with standard clustering and information extraction techniques. For sufficiently frequent words this method produces satisfactory results. Brown et al. (1992) use a very large amount of data, and a well-founded information theo- retic model to induce large numbers of plausi- ble semantic and syntactic clusters. Both ap- proaches have two flaws: they cannot deal well with ambiguity, though Schfitze addresses this issue partially, and they do not cope well with rare words. Since rare and ambiguous words are very common in natural language, these limita- tions are serious. 3 Context Distributions Whereas earlier methods all share the same ba- sic intuition, i.e. that similar words occur in similar contexts, I formalise this in a slightly different way: each word defines a probability distribution over all contexts, namely the prob- ability of the context given the word. If the context is restricted to the word on either side, I can define the context distribution to be a dis- tribution over all ordered pairs of words: the word before and the word after. The context distribution of a word can be estimated from the observed contexts in a corpus. We can then measure the similarity of words by …

Content truncated to 500 words