A Maximum Entropy Model for Part-Of-Speech Tagging

i A Maximum Entropy Model for Part-Of-Speech Tagging Adwait Ratnaparkhi University of Pennsylvania Dept. of Computer and Information Science adwait~gradient, cis. upenn, edu Abstract This paper presents a statistical model which trains from a corpus annotated with Part-Of- Speech tags and assigns them to previously unseen text with state-of-the-art accuracy(96.6%). The model can be classified as a Maximum Entropy model and simultaneously uses many contextual "features" to predict the POS tag. Furthermore, this paper demonstrates the use of specialized fea- tures to model difficult tagging decisions, discusses the corpus consistency problems discovered during the implementation of these features, and proposes a training strategy that mitigates these problems. Introduction Many natural language tasks require the accurate assignment of Part-Of-Speech (POS) tags to pre- viously unseen text. Due to the availability of large corpora which have been manually annotated with POS information, many taggers use anno- tated text to "learn" either probability distribu- tions or rules and use them to automatically assign POS tags to unseen text. The experiments in this paper were conducted on the Wall Street Journal corpus from the Penn Treebank project(Marcus et al., 1994), although the model can trai~n from any large corpus anno- tated with POS tags. Since most realistic natu- ral language applications must process words that were never seen before in training data, all exper- iments in this paper are conducted on test data that include unknown words. Several recent papers(Brill, 1994, Magerman, 1995) have reported 96.5% tag- ging accuracy on the Wall St. Journal corpus. The experiments in this paper test the hy- pothesis that better use of context will improve the accuracy. A Maximum Entropy model is well-suited for such experiments since it corn- bines diverse forms of contextual information in a principled manner, and does not impose any distributional assumptions on the train- ing data. Previous uses of this model include language modeling(Lau et al., 1993), machine translation(Berger et al., 1996), prepositional phrase attachment(Ratnaparkhi et al., 1994), and word morphology(Della Pietra et al., 1995). This paper briefly describes the maximum entropy and maximum likelihood properties of the model, features used for POS tagging, and the experi- ments on the Penn Treebank Wall St. Journal corpus. It then discusses the consistency problems discovered during an attempt to use specialized features on the word context. Lastly, the results in this paper are compared to those from previous work on POS tagging. The Probability Model The probability model is defined over 7-/x 7-, where 7t is the set of possible word and tag contexts, or "histories", and T is the set of allowable tags. The model's probability of a history h together with a tag t is defined as: k p(h, t) = Ill -J- jj(h,t) j=l (1) where ~" is a normalization constant, {tt, ~1,..., ak} are the positive model parameters and {fl,..-,fk} are known as "features", where fj(h,t) E {O, 1}. Note that each parameter aj corresponds to a feature fj. Given a sequence of words {wl,..., Wn} and tags {tl,...t,~} as training …

Content truncated to 500 words