An Empirical Study of the Behavior of Active Learning for Word Sense Disambiguation

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 120–127, New York, June 2006. c⃝2006 Association for Computational Linguistics An Empirical Study of the Behavior of Active Learning for Word Sense Disambiguation 1 Jinying Chen, 1 Andrew Schein, 1 Lyle Ungar, 2 Martha Palmer 1 Department of Computer and Information Science University of Pennsylvania Philadelphia, PA, 19104 {jinying,ais,ungar}@cis.upenn.edu 2 Linguistic Department University of Colorado Boulder, CO, 80309 Martha.Palmer@colorado.edu Abstract This paper shows that two uncertainty- based active learning methods, combined with a maximum entropy model, work well on learning English verb senses. Data analysis on the learning process, based on both instance and feature levels, suggests that a careful treatment of feature extraction is important for the active learning to be useful for WSD. The overfitting phenomena that occurred during the active learning process are identified as classic overfitting in machine learning based on the data analysis. 1 Introduction Corpus-based methods for word sense disambiguation (WSD) have gained popularity in recent years. As evidenced by the SENSEVAL exercises (http://www.senseval.org), machine learning models supervised by sense-tagged training corpora tend to perform better on the lexical sample tasks than unsupervised methods. However, WSD tasks typically have very limited amounts of training data due to the fact that creating large-scale high-quality sense-tagged corpora is difficult and time-consuming. Therefore, the lack of sufficient labeled training data has become a major hurdle to improving the performance of supervised WSD. A promising method for solving this problem could be the use of active learning. Researchers use active learning methods to minimize the labeling of examples by human annotators. A decrease in overall labeling occurs because active learners (the machine learning models used in active learning) pick more informative examples for the target word (a word whose senses need to be learned) than those that would be picked randomly. Active learning requires human labeling of the newly selected training data to ensure high quality. We focus here on pool-based active learning where there is an abundant supply of unlabeled data, but where the labeling process is expensive. In NLP problems such as text classification (Lewis and Gale, 1994; McCallum and Nigam, 1998), statistical parsing (Tang et al., 2002), information extraction (Thompson et al., 1999), and named entity recognition (Shen et al., 2004), pool-based active learning has produced promising results. This paper presents our experiments in applying two active learning methods, a min-margin based method and a Shannon-entropy based one, to the task of the disambiguation of English verb senses. The contribution of our work is not only in demonstrating that these methods work well for the active learning of coarse-grained verb senses, but also analyzing the behavior of the active learning process on two levels: the instance level and the feature level. The analysis suggests that a careful treatment of feature design and feature generation is important for a successful application of active learning to WSD. We also accounted for the overfitting phenomena that occurred in the learning process based …

Content truncated to 500 words