Semantic Lexicon Construction: Learning from Unlabeled Data via Spectral Analysis

Semantic Lexicon Construction: Learning from Unlabeled Data via Spectral Analysis Rie Kubota Ando IBM T.J. Watson Research Center 19 Skyline Dr., Hawthorne, NY 10532 rie1@us.ibm.com Abstract This paper considers the task of automatically collecting words with their entity class labels, starting from a small number of labeled ex- amples (‘seed’ words). We show that spec- tral analysis is useful for compensating for the paucity of labeled examples by learning from unlabeled data. The proposed method signif- icantly outperforms a number of methods that employ techniques such as EM and co-training. Furthermore, when trained with 300 labeled examples and unlabeled data, it rivals Naive Bayes classifiers trained with 7500 labeled ex- amples. 1 Introduction Entity detection plays an important role in information extraction systems. Whether entity recognizers employ machine learning techniques or rule-based approaches, it is useful to have a gazetteer of words1 that reliably sug- gest target entity class membership. This paper considers the task of generating such gazetteers from a large unan- notated corpus with minimal manual effort. Starting from a small number of labeled examples (seeds), e.g., � “car”, “plane”, “ship” � labeled as vehicles, we seek to automat- ically collect more of these. This task is sometimes called the semi-automatic con- struction of semantic lexicons, e.g. (Riloff and Shepherd, 1997; Roark and Charniak, 1998; Thelen and Riloff, 2002; Phillips and Riloff, 2002). A common trend in prior studies is bootstrapping, which is an iterative pro- cess to collect new words and regard the words newly collected with high confidence as additional labeled ex- amples for the next iteration. The aim of bootstrapping is to compensate for the paucity of labeled examples. However, its potential danger is label ‘contamination’ — namely, wrongly (automatically) labeled examples may 1Our argument in this paper holds for relatively small lin- guistic objects including words, phrases, collocations, and so forth. For simplicity, we refer to words. misdirect the succeeding iterations. Also, low frequency words are known to be problematic. They do not pro- vide sufficient corpus statistics (e.g., how frequently the word occurs as the subject of “said”), for adequate label prediction. By contrast, we focus on improving feature vector rep- resentation for use in standard linear classifiers. To coun- teract data sparseness, we employ subspace projection where subspaces are derived by singular value decompo- sition (SVD). In this paper, we generally call such SVD- based subspace construction spectral analysis. Latent Semantic Indexing (LSI) (Deerwester et al., 1990) is a well-known application of spectral analysis to word-by-document matrices. Formal analyses of LSI were published relatively recently, e.g., (Papadimitriou et al., 2000; Azar et al., 2001). Ando and Lee (2001) show the factors that may affect LSI’s performance by analyzing the conditions under which the LSI subspace approximates an optimum subspace. Our theoretical ba- sis is partly derived from this analysis. In particular, we replace the abstract notion of ‘optimum subspace’ with a precise definition of a subspace useful for our task. The essence of spectral analysis is to capture the most prominently observed …

Content truncated to 500 words