Augmented Mixture Models for Lexical Disambiguation

Augmented Mixture Models for Lexical Disambiguation Silviu Cucerzan and David Yarowsky Department of Computer Science and Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218, USA {silviu,yarowsky}@cs.jhu.edu Abstract This paper investigates several augmented mixture models that are competitive alternatives to standard Bayesian models and prove to be very suitable to word sense disambiguation and related classifica- tion tasks. We present a new classification correc- tion technique that successfully addresses the prob- lem of under-estimation of infrequent classes in the training data. We show that the mixture models are boosting-friendly and that both Adaboost and our original correction technique can improve the re- sults of the raw model significantly, achieving state- of-the-art performance on several standard test sets in four languages. With substantially different out- put to Naïve Bayes and other statistical methods, the investigated models are also shown to be effective participants in classifier combination. 1 Introduction The focus tasks of this paper are two re- lated problems in lexical ambiguity resolution: Word Sense Disambiguation (WSD) and Context- Sensitive Spelling Correction (CSSC). Word Sense Disambiguation has a long history as a computational task (Kelly and Stone, 1975), and the field has recently supported large-scale interna- tional system evaluation exercises in multiple lan- guages (SENSEVAL-1, Kilgarriff and Palmer (2000), and SENSEVAL-2, Edmonds and Cotton (2001)). General purpose Spelling Correction is also a long-standing task (e.g. McIlroy, 1982), tradi- tionally focusing on resolving typographical errors such as transposition and deletion to find the clos- est “valid” word (in a dictionary or a morpholog- ical variant), typically ignoring context. Yet Ku- kich (1992) observed that about 25-50% of the spelling errors found in modern documents are ei- ther context-inappropriate misuses or substitutions of valid words (such as principal and principle) which are not detected by traditional spelling cor- rectors. Previous work has addressed the problem of CSSC from a machine learning perspective, in- cluding Bayesian and Decision List models (Gold- ing, 1995), Winnow (Golding and Roth, 1996) and Transformation-Based Learning (Mangu and Brill, 1997). Generally, both tasks involve the selection be- tween a relatively small set of alternatives per key- word (e.g. sense id’s such as church/BUILDING and church/INSTITUTION or commonly confused spellings such as quiet and quite), and are dependent on local and long-distance collocational and syntac- tic patterns to resolve between the set of alterna- tives. Thus both tasks can share a common feature space, data representation and algorithm infrastruc- ture. We present a framework of doing so, while in- vestigating the use of mixture models in conjunction with a new error-correction technique as competi- tive alternatives to Bayesian models. While several authors have observed the fundamental similarities between CSSC and WSD (e.g. Berleant, 1995 and Roth, 1998), to our knowledge no previous com- parative empirical study has tackled these two prob- lems in a single unified framework. 2 Problem Formulation. Feature Space The problem of lexical disambiguation can be mod- eled as a classification task, in which each in- stance of the word to be disambiguated (target …

Content truncated to 500 words