Multilingual and Multimodal Topic Modelling with Pretrained Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 4037–4048 October 12–17, 2022. 4037 Multilingual and Multimodal Topic Modelling with Pretrained Embeddings Elaine Zosa and Lidia Pivovarova Department of Computer Science University of Helsinki Helsinki, Finland firstname.lastname@helsinki.fi Abstract This paper presents M3L-Contrast—a novel multimodal multilingual (M3L) neural topic model for comparable data that maps texts from multiple languages and images into a shared topic space. Our model is trained jointly on texts and images and takes advantage of pre- trained document and image embeddings to abstract the complexities between different lan- guages and modalities. As a multilingual topic model, it produces aligned language-specific topics and as multimodal model, it infers tex- tual representations of semantic concepts in im- ages. We demonstrate that our model is compet- itive with a zero-shot topic model in predicting topic distributions for comparable multilingual data and significantly outperforms a zero-shot model in predicting topic distributions for com- parable texts and images. We also show that our model performs almost as well on unaligned embeddings as it does on aligned embeddings. 1 Introduction Topic modelling is an unsupervised method initially designed for text data that extracts latent themes in documents through the co-occurrence statistics of the words in the documents. In most probabilis- tic topic models, a topic is a distribution over a vocabulary and a document is a distribution over topics (Blei et al., 2003). Multilingual topic mod- els extend basic topic models for multilingual data by jointly training on multiple languages (Mimno et al., 2009; Hao and Paul, 2018). These mod- els learn aligned language-specific topics and have been used in different cross-lingual applications such as multilingual news clustering (De Smet and Moens, 2009) and comparing discourses from dif- ferent cultures in news and social media (Shi et al., 2016; Gutiérrez et al., 2016). Most topic models are designed for textual data but there is also a rich body of work on applying topic modelling to images resulting in multimodal topic models (Barnard et al., 2003; Feng and Lap- ata, 2010; Roller and Im Walde, 2013). These mod- els use natural language supervision to improve the semantic representation of images. Augmenting topical information from text with topic informa- tion from visual inputs also produces better seman- tic representations of words. Neural topic models have been proposed to im- prove on classical topic models and have resulted in models that are more computationally efficient and produces more coherent topics (Srivastava and Sut- ton, 2017). Moreover, the neural topic modelling framework has given rise to models that take advan- tage of information from external sources such as word embeddings (Dieng et al., 2020) and contex- tualised language models (Bianchi et al., 2021a,b; Hoyle et al., 2020; Mueller and Dredze, 2021). In this work, we present a novel neural multilin- gual and multimodal topic model that takes advan- tage of pretrained document and image embeddings to abstract the complexities between languages and modalities. Our work is based on the contextual- ized topic model (CTM, Bianchi et …

Content truncated to 500 words