Categorizing Unknown Words: Using Decision Trees to Identify Names and Misspellings

Categorizing Unknown Words: Using Decision Trees to Identify Names and Misspellings Janine Toole Natural Language Laboratory Department of Computing Science Simon Fraser University Burnaby, BC, Canada VSA IS6 toole@cs.sfu.ca Abstract This paper introduces a system for categorizing un- known words. The system is based on a multi- component architecture where each component is re- sponsible for identifying one class of unknown words. The focus of this paper is the components that iden- tify names and spelling errors. Each component uses a decision tree architecture to combine multiple types of evidence about the unknown word. The sys- tem is evaluated using data from live closed captions - a genre replete with a wide variety of unknown words. 1 Introduction In any real world use, a Natural Language Process- ing (NLP) system will encounter words that are not in its lexicon, what we term 'unknown words'. Un- known words are problematic because a NLP system will perform well only if it recognizes the words that it is meant to analyze or translate: the more words a system does not recognize the more the system's per- formance will degrade. Even when unknown words are infrequent, they can have a disproportionate ef- fect on system quality. For example, Min (1996) found that while only 0.6% of words in 300 e-mails were misspelled, this meant that 12% of the sen- tences contained an error (discussed in (Min and Wilson, 1998)). Words may be unknown for many reasons: the word may be a proper name, a misspelling, an ab- breviation, a number, a morphological variant of a known word (e.g. recleared), or missing from the dictionary. The first step in dealing with unknown words is to identify the class of the unknown word; whether it is a misspelling, a proper name, an ab- breviation etc. Once this is known, the proper ac- tion can be taken, misspellings can be corrected, ab- breviations can be expanded and so on, as deemed necessary by the particular text processing applica- tion. In this paper we introduce a system for cat- egorizing unknown words. The system is based on a multi- component architecture where each compo- nent is responsible for identifying one category of unknown words. The main focus of this paper is the components that identify names and spelling errors. Both components use a decision tree architecture to combine multiple types of evidence about the un- known word. Results from the two components are combined using a weighted voting procedure. The system is evaluated using data from live closed cap- tions - a genre replete with a wide variety of un- known words. This paper is organized as follows. In section 2 we outline the overall architecture of the unknown word categorizer. The name identifier and the mis- spelling identifier are introduced in section 3. Perfor- mance and evaluation issues are discussed in section 4. Section 5 considers portability issues. Section 6 compares the current system with relevant preced- ing research. Concluding comments can be found in section 6. …

Content truncated to 500 words