Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 783–790, Prague, June 2007. c⃝2007 Association for Computational Linguistics Active Learning for Word Sense Disambiguation with Methods for Addressing the Class Imbalance Problem Jingbo Zhu University of Southern California Information Sciences Institute Northeastern University, P.R.China Natural Language Processing Lab Zhujingbo@mail.neu.edu.cn Eduard Hovy University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695 hovy@isi.edu Abstract In this paper, we analyze the effect of resampling techniques, including under- sampling and over-sampling used in active learning for word sense disambiguation (WSD). Experimental results show that un- der-sampling causes negative effects on ac- tive learning, but over-sampling is a rela- tively good choice. To alleviate the within- class imbalance problem of over-sampling, we propose a bootstrap-based over- sampling (BootOS) method that works bet- ter than ordinary over-sampling in active learning for WSD. Finally, we investigate when to stop active learning, and adopt two strategies, max-confidence and min-error, as stopping conditions for active learning. According to experimental results, we sug- gest a prediction solution by considering max-confidence as the upper bound and min-error as the lower bound for stopping conditions. 1 Introduction Word sense ambiguity is a major obstacle to accu- rate information extraction, summarization, and machine translation (Ide and Veronis, 1998). In recent years, a variety of techniques for machine learning algorithms have demonstrated remarkable performance for automated word sense disam- biguation (WSD) (Chan and Ng, 2006; Dagan et. al., 2006; Xue et. al., 2006; Kohomban and Lee. 2005; Dang and Palmer, 2005), when enough la- beled training data is available. However, creating a large sense-tagged corpus is very expensive and time-consuming, because these data have to be an- notated by human experts. Among the techniques to solve the knowledge bottleneck problem, active learning is a promising way (Lewis and Gale, 1994; McCallum and Ni- gram, 1998). The purpose of active learning is to minimize the amount of human labeling effort by having the system automatically select for human annotation the most informative unannotated case. In real-world data, the distribution of the senses of a word is often very skewed. Some studies re- ported that simply selecting the predominant sense provides superior performance, when a highly skewed sense distribution and insufficient context exist (Hoste et al., 2001; McCarthy et. al., 2004). The data set is imbalanced when at least one of the senses is heavily underrepresented compared to the other senses. In general, a WSD classifier is de- signed to optimize overall accuracy without taking into account the class imbalance distribution in a real-world data set. The result is that the classifier induced from imbalanced data tends to over-fit the predominant class and to ignore small classes (Japkowicz and Stephen, 2002). Recently, much work has been done in addressing the class imbalance problem, reporting that resampling methods such as over-sampling and under- sampling are useful in supervised learning with imbalanced data sets to induce more effective classifiers (Estabrooks et al., …

Content truncated to 500 words