Improving Probabilistic Latent Semantic Analysis with Principal Component Analysis

Improving Probabilistic Latent Semantic Analysis with Principal Component Analysis Ayman Farahat Palo Alto Research Center 3333 Coyote Hill Road Palo Alto, CA 94304 ayman.farahat@gmail.com Francine Chen Palo Alto Research Center 3333 Coyote Hill Road Palo Alto, CA 94304 chen@fxpal.com Abstract Probabilistic Latent Semantic Analysis (PLSA) models have been shown to pro- vide a better model for capturing poly- semy and synonymy than Latent Seman- tic Analysis (LSA). However, the param- eters of a PLSA model are trained using the Expectation Maximization (EM) algo- rithm, and as a result, the trained model is dependent on the initialization values so that performance can be highly variable. In this paper we present a method for using LSA analysis to initialize a PLSA model. We also investigated the performance of our method for the tasks of text segmenta- tion and retrieval on personal-size corpora, and present results demonstrating the effi- cacy of our proposed approach. 1 Introduction In modeling a collection of documents for infor- mation access applications, the documents are of- ten represented as a “bag of words”, i.e., as term vectors composed of the terms and corresponding counts for each document. The term vectors for a document collection can be organized into a term by document co-occurrence matrix. When di- rectly using these representations, synonyms and polysemous terms, that is, terms with multiple senses or meanings, are not handled well. Meth- ods for smoothing the term distributions through the use of latent classes have been shown to im- prove the performance of a number of information access tasks, including retrieval over smaller col- lections (Deerwester et al., 1990), text segmenta- tion (Brants et al., 2002), and text classification (Wu and Gunopulos, 2002). The Probabilistic Latent Semantic Analysis model (PLSA) (Hofmann, 1999) provides a prob- abilistic framework that attempts to capture poly- semy and synonymy in text for applications such as retrieval and segmentation. It uses a mixture decomposition to model the co-occurrence data, and the probabilities of words and documents are obtained by a convex combination of the aspects. The mixture approximation has a well defined probability distribution and the factors have a clear probabilistic meaning in terms of the mixture com- ponent distributions. The PLSA model computes the relevant proba- bility distributions by selecting the model parame- ter values that maximize the probability of the ob- served data, i.e., the likelihood function. The stan- dard method for maximum likelihood estimation is the Expectation Maximization (EM) algorithm. For a given initialization, the likelihood function increases with EM iterations until a local maxi- mum is reached, rather than a global maximum, so that the quality of the solution depends on the initialization of the model. Additionally, the likeli- hood values across different initializations are not comparable, as we will show. Thus, the likelihood function computed over the training data cannot be used as a predictor of model performance across different models. Rather than trying to predict the best perform- ing model from a set of models, in this paper we focus on finding …

Content truncated to 500 words