Cross-lingual Contextualized Topic Models with Zero-shot Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1676–1683 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1676 Cross-lingual Contextualized Topic Models with Zero-shot Learning Federico Bianchi Bocconi University Via Sarfatti 25, 20136 Milan, Italy f.bianchi@unibocconi.it Silvia Terragni University of Milano-Bicocca Viale Sarca 336, 20126 Milan, Italy s.terragni4@campus.unimib.it Dirk Hovy Bocconi University Via Sarfatti 25, 20136 Milan, Italy dirk.hovy@unibocconi.it Debora Nozza Bocconi University Via Sarfatti 25, 20136 Milan, Italy debora.nozza@unibocconi.it Elisabetta Fersini University of Milano-Bicocca Viale Sarca 336, 20126 Milan, Italy elisabetta.fersini@unimib.it Abstract Many data sets (e.g., reviews, forums, news, etc.) exist parallelly in multiple languages. They all cover the same content, but the lin- guistic differences make it impossible to use traditional, bag-of-word-based topic models. Models have to be either single-language or suffer from a huge, but extremely sparse vo- cabulary. Both issues can be addressed by transfer learning. In this paper, we introduce a zero-shot cross-lingual topic model. Our model learns topics on one language (here, English), and predicts them for unseen doc- uments in different languages (here, Italian, French, German, and Portuguese). We evalu- ate the quality of the topic predictions for the same document in different languages. Our re- sults show that the transferred topics are co- herent and stable across languages, which sug- gests exciting future research directions. 1 Introduction Topic models (Blei et al., 2003; Blei, 2012) allow us to find the main themes and overarching tropes in textual data. However, traditional methods are language-specific and cannot be used in a trans- ferable manner. They rely on a fixed vocabulary specific to the training language. Therefore, currently available topic models suf- fer from two limitations: (i) they cannot handle unknown words by default, and (ii) they cannot eas- ily be applied to other languages - except the one in the training data - since the vocabulary would not match. Training on several languages together, though, results in a vocabulary so vast that it creates problems with parameter size, search, and overfit- ting (Boyd-Graber et al., 2014). Traditional topic modeling provides methods to extract meaningful word distributions from “unstructured” text but re- quires language-specific bag-of-words (BoW) rep- resentations (Boyd-Graber and Blei, 2009; Jagarla- mudi and Daum´e, 2010). A cross-lingual setup proves ideal for transfer learning: provided that the gist of topics is the same across languages, we can learn this gist on texts in one language and then apply it to others. This setup is zero-shot learning: we train a model on one language and test it on several other languages to which the model had no access during training. To this end, we need to leverage external in- formation to support the topic modeling task. In- deed, topic models have often gained significant advantages from introducing external knowledge, e.g., document relationships (Yang et al., 2015; Wang et al., 2020; Terragni et al., 2020a,b) and word embeddings (Nozza et al., 2016; Li et al., 2016; Zhao et al., 2017; Dieng et al., 2020). Recently, pre-trained …

Content truncated to 500 words