Part-of-Speech Induction From Scratch

PART-OF-SPEECH INDUCTION FROM SCRATCH Hinrich Schiitze Center for the Study of Language and Information Ventura Hall Stanford, CA 94305-4115 schuetze~csli.stanford.edu Abstract This paper presents a method for inducing the parts of speech of a language and part- of-speech labels for individual words from a large text corpus. Vector representations for the part-of-speech of a word are formed from entries of its near lexical neighbors. A dimen- sionality reduction creates a space represent- ing the syntactic categories of unambiguous words. A neural net trained on these spa- tial representations classifies individual con- texts of occurrence of ambiguous words. The method classifies both ambiguous and unam- biguous words correctly with high accuracy. INTRODUCTION Part-of-speech information about individual words is necessary for any kind of syntactic and higher level processing of natural language. While it is easy to obtain lists with part of speech labels for frequent English words, such information is not available for less common languages. Even for En- glish, a categorization of words that is tailored to a particular genre may be desired. Finally, there are rare words that need to be categorized even if fre- quent words are covered by an available electronic dictionary. This paper presents a method for inducing the parts of speech of a language and part-of-speech labels for individual words from a large text cor- pus. Little, if any, language-specific knowledge is used, so that it is applicable to any language in principle. Since the part-of-speech representations are derived from the corpus, the resulting catego- rization is highly text specific and doesn't contain categories that are inappropriate for the genre in question. The method is efficient enough for vo- cabularies of tens of thousands of words thus ad- dressing the problem of coverage. The problem of how syntactic categories can be induced is also of theoretical interest in language acquisition and learnability. Syntactic category information is part of the basic knowledge about language that children must learn before they can acquire more complicated structures. It has been claimed that "the properties that the child can detect in the input - such as the serial positions and adjacency and co-occurrence relations among words - are in general linguistically irrelevant." (Pinker 1984) It will be shown here that relative position of words with respect to each other is suf- ficient for learning the major syntactic categories. In the first part of the derivation, two iterations of a massive linear approximation of cooccurrence counts categorize unambiguous words. Then a neural net trained on these words classifies indi- vidual contexts of occurrence of ambiguous words. An evaluation suggests that the method classi- fies both ambiguous and unambiguous words cor- rectly. It differs from previous work in its effi- ciency and applicability to large vocabularies; and in that linguistic knowledge is only used in the very last step so that theoretical assumptions that don't hold for a language or sublanguage have min- imal influence on the classification. The next two sections describe the linear ap- proximation and a birecurrent neural …

Content truncated to 500 words