Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3054–3068 June 6–11, 2021. ©2021 Association for Computational Linguistics 3054 Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling Aaron Mueller and Mark Dredze Department of Computer Science Johns Hopkins University amueller@jhu.edu, mdredze@cs.jhu.edu Abstract Neural topic models can augment or replace bag-of-words inputs with the learned represen- tations of deep pre-trained transformer-based word prediction models. One added benefit when using representations from multilingual models is that they facilitate zero-shot polylin- gual topic modeling. However, while it has been widely observed that pre-trained embed- dings should be fine-tuned to a given task, it is not immediately clear what supervision should look like for an unsupervised task such as topic modeling. Thus, we propose several meth- ods for fine-tuning encoders to improve both monolingual and zero-shot polylingual neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing a new topic clas- sification task, integrating the topic classifica- tion objective directly into topic model train- ing, and continued pre-training. We find that fine-tuning encoder representations on topic classification and integrating the topic classi- fication task directly into topic modeling im- proves topic quality, and that fine-tuning en- coder representations on any task is the most important factor for facilitating cross-lingual transfer. 1 Introduction Topic models (Blei et al., 2003) are widely used across numerous disciplines to study large corpora (Boyd-Graber et al., 2017). These data-driven mod- els discover salient themes and semantic clusters without any supervision. Monolingual topic mod- els are language-agnostic but do not align topics across languages, as they have a fixed language- specific vocabulary which cannot be aligned cross- lingually after training. Polylingual topic models (Mimno et al., 2009), however, enable users to consider multilingual corpora, and to discover and align topics across languages. Recent work has demonstrated the effectiveness of deep transformer-based language models to en- code text documents for a wide variety of applica- tions (Xia et al., 2020). Furthermore, when trained on multilingual corpora, they have been able to dis- cover cross-lingual alignments despite the lack of explicit cross-lingual links (Wu and Dredze, 2019). Models such as multilingual BERT (mBERT; De- vlin et al., 2018) or XLM-RoBERTa (XLM-R; Con- neau et al., 2019) can produce a representation of text in a shared subspace across multiple input lan- guages, suitable for both monolingual and multilin- gual settings, including zero-shot language transfer (Pires et al., 2019). Simultaneously, topic models have increasingly incorporated neural components. This has included inference networks which learn representations of the input document (Miao et al., 2017; Srivastava and Sutton, 2017) that improve over using bags of words directly, as well as replacing bags of words with contextual representations. In particu- lar, the latter allows topic models to benefit from pre-training on large corpora. For example, con- textualized topic models (CTMs) (Bianchi et al., 2020a) use autoencoded contextual sentence repre- sentations of input documents. An intriguing advantage of using encoders in topic models is …

Content truncated to 500 words