Analyzing Bayesian Crosslingual Transfer in Topic Models

Proceedings of NAACL-HLT 2019, pages 1551–1565 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1551 Analyzing Bayesian Crosslingual Transfer in Topic Models Shudong Hao Boulder, CO shudonghao@gmail.com Michael J. Paul Information Science University of Colorado Boulder, CO mpaul@colorado.edu Abstract We introduce a theoretical analysis of crosslin- gual transfer in probabilistic topic models. By formulating posterior inference through Gibbs sampling as a process of language transfer, we propose a new measure that quantifies the loss of knowledge across languages during this process. This measure enables us to derive a PAC-Bayesian bound that elucidates the fac- tors affecting model quality, both during train- ing and in downstream applications. We pro- vide experimental validation of the analysis on a diverse set of five languages, and discuss best practices for data collection and model design based on our analysis. 1 Introduction Crosslingual learning is an important area of nat- ural language processing that has driven appli- cations including text mining in multiple lan- guages (Ni et al., 2009; Smet and Moens, 2009), cultural difference detection (Guti´errez et al., 2016), and various linguistic studies (Shutova et al., 2017; Barrett et al., 2016). Crosslin- gual learning methods generally extend mono- lingual algorithms by using various multilin- gual resources. In contrast to traditional high-dimensional vector space models, modern crosslingual models tend to rely on learning low- dimensional word representations that are more efficient and generalizable. A popular approach to representation learn- ing comes from the word embedding commu- nity, in which words are represented as vectors in an embedding space shared by multiple lan- guages (Ruder et al., 2018; Faruqui and Dyer, 2014; Klementiev et al., 2012). Another di- rection is from the topic modeling community, where words are projected into a probabilistic topic space (Ma and Nasukawa, 2017; Jagarla- mudi and III, 2010). While formulated differently, both types of models apply the same principles— low-dimensional vectors exist in a shared crosslin- gual space, wherein vector representations of sim- ilar concepts across languages (e.g., “dog” and “hund”) should be nearby in the shared space. To enable crosslingual representation learning, knowledge is transferred from a source language to a target language, so that representations have similar values across languages. In this study, we will focus on probabilistic topic models, and “knowledge” refers to a word’s probability distri- bution over topics. Little is known about the char- acteristics of crosslingual knowledge transfer in topic models, and thus this paper provides an anal- ysis, both theoretical and empirical, of crosslin- gual transfer in multilingual topic models. 1.1 Background and Contributions Multilingual Topic Models Given a multilin- gual corpus D(1,...,L) in languages ℓ = 1, . . . , L as inputs, a multilingual topic model learns K topics. Each multilingual topic k(1,...,L) (k = (1, . . . , K), is defined as an L-dimensional tuple ϕ(1) k , . . . , ϕ(L) k ) , where ϕ(ℓ) k is a multinomial dis- tribution over the vocabulary V (ℓ) in language ℓ. From a …

Content truncated to 500 words