BERT-Flow-VAE: A Weakly-supervised Model for Multi-Label Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 1203–1220 October 12–17, 2022. 1203 BERT-Flow-VAE: A Weakly-supervised Model for Multi-Label Text Classification Ziwen Liu University College London z.liu.19@ucl.ac.uk Scott Allan Orr University College London scott.orr@ucl.ac.uk Josep Grau-Bové University College London josep.grau.bove@ucl.ac.uk Abstract Multi-label Text Classification (MLTC) is the task of categorizing documents into one or more topics. Considering the large vol- umes of data and varying domains of such tasks, fully-supervised learning requires man- ually fully annotated datasets which is costly and time-consuming. In this paper, we propose BERT-Flow-VAE (BFV), a Weakly- Supervised Multi-Label Text Classification (WSMLTC) model that reduces the need for full supervision. This new model: (1) pro- duces BERT sentence embeddings and cali- brates them using a flow model, (2) generates an initial topic-document matrix by averaging results of a seeded sparse topic model and a textual entailment model that only require surface name of topics and 4-6 seed words per topic, and (3) adopts a VAE framework to reconstruct the embeddings under the guid- ance of the topic-document matrix. Finally, (4) it uses the means produced by the encoder model in the VAE architecture as predictions for MLTC. Experimental results on 6 multi- label datasets show that BFV can substantially outperform other baseline WSMLTC models in key metrics and achieve approximately 84% performance of a fully-supervised model. 1 Introduction As vast numbers of written comments are posted daily on social media and e-commerce platforms, there is an increasing demand for methods that efficiently and effectively extract useful informa- tion from this unstructured text data. One of the methods to analyze this unstructured text data is to classify them into organized categories. This can be considered as a Multi-label Text Classifica- tion (MLTC) task since a single data may contain multiple non-mutually-exclusive topics (aspects). There are a range of relevant applications of this task such as categorizing movies by genres (Hoang, 2018), multi-label sentiment analysis (Almeida et al., 2018) and multi-label toxicity identification (Gunasekara and Nejadgholi, 2018). Fully-supervised learning methods are undesir- able for this task, because of the diversity of do- mains of application and cost of manual labelling (Brody and Elhadad, 2010). Seeded topic mod- els, such as SeededLDA and CorEx (Jagarlamudi et al., 2012; Gallagher et al., 2017), where users can designate seed words as a prior to guide the models to find topics of interest, can be seen as a Weakly-Supervised Multi-Label Text Classifica- tion (WSMLTC) method. Nevertheless, as these models are mainly statistical models based on bag- of-words representation, they fail to fully exploit key sentence elements such as context and word positions. In contrast, large pre-trained language models such as BERT and GPT-3 (Devlin et al., 2018; Brown et al., 2020) produce contextualized embeddings for each word in a sentence, which has afforded them great success in the NLP field (Minaee et al., 2021; Ethayarajh, 2019). Recently, prompt-based Few-Shot Learning (FSL) and Zero-Shot Learning (ZSL) methods (Yin et al., 2019, 2020; Gao et al., 2020) that take advan- tage of …

Content truncated to 500 words