Tagging Unknown Words with Raw Text Features

Proceedings of the Australasian Language Technology Workshop 2005, pages 32–39, Sydney, Australia, December 2005. Tagging Unknown Words with Raw Text Features David Vadas and James R. Curran School of Information Technologies University of Sydney NSW 2006, Australia {dvadas1,james}@it.usyd.edu.au Abstract Processing unknown words is disproportionately important because of their high information con- tent. It is crucial in domains with specialist vocab- ularies where relevant training material is scarce, for example: biological text. Unknown word pro- cessing often begins with Part of Speech (POS) tag- ging, where accuracy is typically 10% worse than on known words. We demonstrate that features extracted from large raw text corpora can significantly increase accuracy on unknown words. These features supply a large part of what we are missing with unknown words: context information about how the word is used. We describe a Maximum Entropy modelling approach which uses real-valued features to represent unan- notated contextual information. Our initial experi- ments with real-valued features have resulted in an increased accuracy from 87.39% to 88.85% on un- known words. 1 Introduction Part of Speech (POS) tagging involves assigning ba- sic grammatical classes such as verb, noun and ad- jective to individual words, and is a fundamental step in many Natural Language Processing (NLP) tasks. The tags it assigns are used in other process- ing tasks such as chunking and parsing, as well as in more complex systems for question answering and automatic summarisation. All POS taggers suffer a significant decrease in accuracy on unknown words, that is, words that have not been previously seen in the annotated training set. A loss of up to 10% is typical for most POS tag- gers e.g. Brill (1994) and Ratnaparkhi (1996). This decreased accuracy has a flow on effect for the accu- racy of both following POS tags and later processes which utilise them. Unknown words also occur a significant amount of the time, ranging from 2% – 5% (Mikheev, 1997), depending on the training and test corpus. These figures are much higher for domains with large spe- cialist vocabularies, for example biological text. We improve the performance of a Maximum En- tropy POS tagger by implementing features with non-negative real values. Although Maximum En- tropy is typically described assuming binary-valued features, they are not in fact required to be binary valued. The only limitations come from the optimi- sation algorithm. For example, the Generalised Iter- ative Scaling (Darroch and Ratcliff, 1972) algorithm used in these experiments imposes a non-negativity constraint on feature values. Real-valued features can encapsulate contextual information extracted from around unknown word occurrences in an unannotated corpus. Using a large corpus is important because this increases the relia- bility of the real-values. By looking at the surround- ing words, we can formulate constraints on what POS tag(s) could be assigned. This can be seen in the sentence below: (1) The frub house is up on the hill Here, frub is the unknown word which, as com- petent speakers of the language, we can surmise is probably a noun …

Content truncated to 500 words