An Empirical Study on Crosslingual Transfer in Probabilistic Topic Models

An Empirical Study on Crosslingual Transfer in Probabilistic Topic Models Shudong Hao Bard College at Simon’s Rock Division of Science, Mathematics, and Computing shao@simons-rock.edu Michael J. Paul University of Colorado Department of Information Science mpaul@colorado.edu Probabilistic topic modeling is a common first step in crosslingual tasks to enable knowledge transfer and extract multilingual features. Although many multilingual topic models have been developed, their assumptions about the training corpus are quite varied, and it is not clear how well the different models can be utilized under various training conditions. In this article, the knowledge transfer mechanisms behind different multilingual topic models are systematically studied, and through a broad set of experiments with four models on ten languages, we provide empirical insights that can inform the selection and future development of multilingual topic models. 1. Introduction Popularized by Latent Dirichlet Allocation (Blei, Ng, and Jordan 2003), probabilistic topic models have been an important tool for analyzing large collections of texts (Blei 2012, 2018). Their simplicity and interpretability make topic models popular for many natural language processing tasks, such as discovery of document networks (Chen et al. 2013; Chang and Blei 2009) and authorship attribution (Seroussi, Zukerman, and Bohnert 2014). Topic models take a corpus D as input, where each document d ∈ D is usually represented as a sparse vector in a vocabulary space, and project these documents to a lower-dimensional topic space. In this sense, topic models are often used as a dimensionality reduction technique to extract representative and human-interpretable features. Text collections, however, are often not in a single language, and thus there has been a need to generalize topic models from monolingual to multilingual settings. Given a corpus D(1,...,L) in languages ℓ ∈ {1, . . . , L}, multilingual topic models learn topics in Submission received: 11 October 2018; revised version received: 22 November 2019; accepted for publication: 25 November 2019. https://doi.org/10.1162/COLI a 00369 © 2020 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 46, Number 1 each of the languages. From a human’s view, each topic should be related to the same theme, even if the words are not in the same language (Figure 1(b)). From a machine’s view, the word probabilities within a topic should be similar across languages, such that the low-dimensional representation of documents is not dependent on the language. In other words, the topic space in multilingual topic models is language agnostic (Figure 1(a)). This article presents two major contributions to multilingual topic models. We first provide an alternative view of multilingual topic models by explicitly formulating a crosslingual knowledge transfer process during posterior inference (Section 3). Based on this analysis, we unify different multilingual topic models by defining a function called the transfer operation. This function provides an abstracted view of the knowledge transfer mechanism behind these models, while enabling further generalizations and improvements. Using this formulation, we analyze several existing multilingual topic models (Section 4). Second, in our experiments we …

Content truncated to 500 words