BART-TL: Weakly-Supervised Topic Label Generation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1418–1425 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1418 BART-TL: Weakly-Supervised Topic Label Generation Cristian Popa Universitatea Politehnica of Bucharest cristian.viorel.popa@gmail.com Traian Rebedea Universitatea Politehnica of Bucharest traian.rebedea@upb.ro Abstract We propose a novel solution for assigning la- bels to topic models by using multiple weak la- belers. The method leverages generative trans- formers to learn accurate representations of the most important topic terms and candidate labels. This is achieved by fine-tuning pre- trained BART models on a large number of po- tential labels generated by state of the art non- neural models for topic labeling, enriched with different techniques. The proposed BART-TL model is able to generate valuable and novel labels in a weakly-supervised manner and can be improved by adding other weak labelers or distant supervision on similar tasks. 1 Introduction As topic modeling has been used for unsupervised exploration of large text corpora, several topic label- ing approaches have been proposed. These range from heuristic-based methods (Mei et al., 2007; Gourru et al., 2018) that focus on the underlying topic distributions to newer methods that use word embeddings (Bhatia et al., 2016). Supervised topic labeling methods (Lau et al., 2011; Bhatia et al., 2016) typically use annotator data with the qual- ity of the labels to train a more accurate ranker than the unsupervised counterpart. Deep learn- ing approaches, which gained quick popularity in NLP, are starting to be used for solving this task as well (Sorodoc et al., 2017; Alokaili et al., 2020). Recently, transformer models pre-trained on very large amounts of data achieved impressive results on a lot of downstream NLP tasks using fewer resources than previously necessary. We intro- duce a method of performing a weakly-supervised fine-tuning on these models pre-trained on English data in order to obtain human-comprehensible and meaningful topic labels. We also provide a qual- ity evaluation of the model-generated labels, in addition to an analysis of the contribution gained from using this approach that we ultimately refer to as BART-TL, inspired by the name of the original transformer architecture. 2 Related Work Topic modeling is a popular unsupervised method for exploring large corpora of documents. Top- ics are represented as distributions over words, while documents as mixtures of topics. Histori- cally, these methods used dimensionality reduction techniques (Deerwester et al., 1990), then migrated to probabilistic-based methods (Hofmann, 1999), with Latent Dirichlet Allocation (Blei et al., 2003) gaining popularity. LDA makes use of variational inference to obtain the distribution matrices. Fur- ther developments include hierarchical (Wang et al., 2011) and online (Hoffman et al., 2010) versions of LDA. While the resulting distributions of topic mod- els are useful for computational purposes, such as measuring the similarity of two documents, these may prove difficult to interpret by humans. Topic labeling aims to solve this issue by computing la- bels for each topic. Historically, this was achieved by establishing a pool of labels and ranking …

Content truncated to 500 words