Investigating Unsupervised Learning for Text Categorization Bootstrapping

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 129–136, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Investigating Unsupervised Learning for Text Categorization Bootstrapping Alfio Gliozzo and Carlo Strapparava ITC-irst Istituto per la Ricerca Scientifica e Tecnologica I-38050 Trento, Italy {gliozzo,strappa}@itc.it Ido Dagan Computer Science Department Bar Ilan University Ramat Gan, Israel dagan@cs.biu.ac.il Abstract We propose a generalized bootstrapping algorithm in which categories are de- scribed by relevant seed features. Our method introduces two unsupervised steps that improve the initial categorization step of the bootstrapping scheme: (i) using La- tent Semantic space to obtain a general- ized similarity measure between instances and features, and (ii) the Gaussian Mixture algorithm, to obtain uniform classification probabilities for unlabeled examples. The algorithm was evaluated on two Text Cate- gorization tasks and obtained state-of-the- art performance using only the category names as initial seeds. 1 Introduction Supervised classification is the task of assigning cat- egory labels, taken from a predefined set of cate- gories (classes), to instances in a data set. Within the classical supervised learning paradigm, the task is approached by providing a learning algorithm with a training data set of manually labeled examples. In practice it is not always easy to apply this schema to NLP tasks. For example supervised systems for Text Categorization (TC) require a large amount of hand labeled texts, while in many applicative cases it is quite difficult to collect the required amounts of hand labeled data. Unlabeled text collections, on the other hand, are in general easily available. An alternative approach is to provide the neces- sary supervision by means of sets of “seeds” of in- tuitively relevant features. Adopting terminology from computability theory, we refer to the stan- dard example-based supervision mode as Exten- sional Learning (EL), as classes are being specified by means of examples of their elements (their ex- tension). Feature-based supervision is referred to as Intensional Learning (IL), as features may often be perceived as describing the intension of a category, such as providing the name or prominent key terms for a category in text categorization. The IL approach reflects on classical rule-based classification methods, where the user is expected to specify exact classification rules that operate in the feature space. Within the machine learning paradigm, IL has been incorporated as a technique for bootstrapping an extensional learning algorithm, as in (Yarowsky, 1995; Collins and Singer, 1999; Liu et al., 2004). This way the user does not need to specify exact classification rules (and fea- ture weights), but rather perform a somewhat sim- pler task of specifying few typical seed features for the category. Given the list of seed features, the bootstrapping scheme consists of (i) preliminary un- supervised categorization of the unlabeled data set based on the seed features, and (ii) training an (ex- tensional) supervised classifier using the automatic classification labels of step (i) as the training data (the second step is possibly reiterated, such as by an Expectation-Maximization schema). The core …

Content truncated to 500 words