Context clustering for Word Sense Disambiguation based on modeling pairwise context similarities

Context Clustering for Word Sense Disambiguation Based on Modeling Pairwise Context Similarities Cheng Niu, Wei Li, Rohini K. Srihari, Huifeng Li, Laurie Crist Cymfony Inc. 600 Essjay Road, Williamsville, NY 14221. USA. {cniu, wei, rohini, hli, lcrist}@cymfony.com Abstract Traditionally, word sense disambiguation (WSD) involves a different context model for each individual word. This paper presents a new approach to WSD using weakly supervised learning. Statistical models are not trained for the contexts of each individual word, but for the similarities between context pairs at category level. The insight is that the correlation regularity between the sense distinction and the context distinction can be captured at category level, independent of individual words. This approach only requires a limited amount of existing annotated training corpus in order to disambiguate the entire vocabulary. A context clustering scheme is developed within the Bayesian framework. A maximum entropy model is then trained to represent the generative probability distribution of context similarities based on heterogeneous features, including trigger words and parsing structures. Statistical annealing is applied to derive the final context clusters by globally fitting the pairwise context similarity distribution. Benchmarking shows that this new approach significantly outperforms the existing WSD systems in the unsupervised category, and rivals supervised WSD systems. 1 Introduction Word Sense Disambiguation (WSD) is one of the central problems in Natural Language Processing. The difficulty of this task lies in the fact that context features and the corresponding statistical distribution are different for each individual word. Traditionally, WSD involves modeling the contexts for each word. [Gale et al. 1992] uses the Naïve Bayes method for context modeling which requires a manually truthed corpus for each ambiguous word. This causes a serious Knowledge Bottleneck. The situation is worse when considering the domain dependency of word senses. To avoid the Knowledge Bottleneck, unsupervised or weakly supervised learning approaches have been proposed. These include the bootstrapping approach [Yarowsky 1995] and the context clustering approach [Schutze 1998]. Although the above unsupervised or weakly supervised learning approaches are less subject to the Knowledge Bottleneck, some weakness exists: i) for each individual keyword, the sense number has to be provided and in the bootstrapping case, seeds for each sense are also required; ii) the modeling usually assumes some form of evidence independency, e.g. the vector space model used in [Schutze 1998] and [Niu et al. 2003]: this limits the performance and its potential enhancement; iii) most WSD systems either use selectional restriction in parsing relations, and/or trigger words which co-occur within a window size of the ambiguous word. We previously at-tempted combining both types of evidence but only achieved limited improvement due to the lack of a proper modeling of information over-lapping [Niu et al. 2003]. This paper presents a new algorithm that addresses these problems. A novel context clustering scheme based on modeling the similarities between pairwise contexts at category level is presented in the Bayesian framework. A generative maximum entropy model is then trained to represent the generative probability distribution of pairwise context similarities based on …

Content truncated to 500 words