Topic Identification and Discovery on Text and Speech

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2377–2387, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Topic Identification and Discovery on Text and Speech Chandler May, Francis Ferraro, Alan McCree, Jonathan Wintrode, Daniel Garcia-Romero, and Benjamin Van Durme Human Language Technology Center of Excellence Johns Hopkins University cjmay@jhu.edu, ferraro@cs.jhu.edu, alan.mccree@jhu.edu, jcwintr@cs.jhu.edu, dgromero@jhu.edu, vandurme@cs.jhu.edu Abstract We compare the multinomial i-vector framework from the speech community with LDA, SAGE, and LSA as fea- ture learners for topic ID on multinomial speech and text data. We also compare the learned representations in their abil- ity to discover topics, quantified by dis- tributional similarity to gold-standard top- ics and by human interpretability. We find that topic ID and topic discovery are competing objectives. We argue that LSA and i-vectors should be more widely con- sidered by the text processing community as pre-processing steps for downstream tasks, and also speculate about speech pro- cessing tasks that could benefit from more interpretable representations like SAGE. 1 Introduction The text processing and speech processing re- search communities have similar problems and goals, but the technical approaches in these two communities develop largely independently. In this paper we compare dimensionality reduction techniques on multinomial language data from the text and speech communities. We consider a multinomial formulation of the i-vector model (hereafter “mi-vector” model) from the speech community (Soufifar et al., 2011), the sparse ad- ditive generative (SAGE) (Eisenstein et al., 2011) and latent Dirichlet allocation (LDA) (Blei et al., 2003b) topic models from the text community, and latent semantic analysis (LSA) (Deerwester et al., 1990). Both the mi-vector model and the SAGE topic model represent a multinomial pa- rameter vector as the softmax of a sum of vectors, one of which is a background vector representing overall word usage in the corpus, and so we might expect mi-vectors and SAGE to produce similar results on real-world data. We evaluate these two recent models and two more conventional mod- els, LDA and LSA (a term describing a class of methods based on the singular value decomposi- tion, or SVD, which is used broadly in both re- search communities). We assess the similarity of mi-vectors and SAGE and expose the strengths and weaknesses of all four learned representations by evaluating them on the supervised task of topic identification (topic ID), depicted in Figure 1. We also evaluate the representations on the unsuper- vised, less easily-measurable task of topic discov- ery. As a proxy for controlled human annotations, we quantify topic discovery performance by dis- tributional similarity to gold-standard topics. We use the bag-of-words multinomial represen- tation of text data, i.e., each document is repre- sented by a vector of counts over the word vo- cabulary. For speech data, we use a modern au- tomatic speech recognition (ASR) system to pro- duce frame-wise triphone state cluster posteriors and we take the sum of these posteriors across all frames in a document to obtain a document- level vector of triphone state cluster soft …

Content truncated to 500 words