Unsupervised Fine-tuning for Text Clustering

Proceedings of the 28th International Conference on Computational Linguistics, pages 5530–5534 Barcelona, Spain (Online), December 8-13, 2020 5530 Unsupervised Fine-tuning for Text Clustering Shaohan Huang, Furu Wei, Lei Cui, Xingxing Zhang, Ming Zhou Microsoft Research, Beijing, China {shaohanh, fuwei, lecu, xizhang, mingzhou}@microsoft.com Abstract Fine-tuning with pre-trained language models (e.g. BERT) has achieved great success in many language understanding tasks in supervised settings (e.g. text classification). However, relatively little work has been focused on applying pre-trained models in unsupervised settings, such as text clustering. In this paper, we propose a novel method to fine-tune pre-trained models unsupervis- edly for text clustering, which simultaneously learns text representations and cluster assignments using a clustering oriented loss. Experiments on three text clustering datasets (namely TREC-6, Yelp, and DBpedia) show that our model outperforms the baseline methods and achieves state- of-the-art results. 1 Introduction Pre-trained language models have shown remarkable progress in many natural language understanding tasks (Radford et al., 2018; Peters et al., 2018; Howard and Ruder, 2018). Especially, BERT (Devlin et al., 2018) applies the fine-tuning approach to achieve ground-breaking performance in a set of NLP tasks. BERT, a deep bidirectional transformer model (Vaswani et al., 2017), utilizes a huge unlabeled data to learn complex features and representations and then fine-tunes its pre-trained model on the downstream tasks with labeled data. Although BERT has achieved great success in many natural language understanding tasks under su- pervised fine-tuning approaches, relatively little work has been focused on applying pre-trained models in unsupervised settings. In this paper, by a case study of text clustering, we investigate how to leverage the pre-trained BERT model and fine-tune it in unsupervised settings, such as text clustering. Previous approaches have made some progress on text clustering using deep neural networks (Min et al., 2018; Aljalbout et al., 2018). Existing deep clustering approaches fall into two categories: two-stage and jointly optimization. Two-stage approach uses deep learning frameworks to learn the representation first and then run clustering algorithms (Chen, 2015; Yang et al., 2017). As the name implies, jointly optimization approaches learn the representations and clustering jointly (Xie et al., 2016; Guo et al., 2017). Inspired by those methods, we can fine-tune pre-trained models by learning text representations and cluster assignments simultaneously. In this paper, we propose a novel method to fine-tune pre-trained models unsupervisedly for text clus- tering. Our model simultaneously learns text representations and cluster assignments by jointly optimiz- ing both the masked language model loss and the clustering oriented loss. The masked language model loss can help learn domain-specific knowledge and guarantee that our model not be misguided such as all-zero vector (Yang et al., 2017). Clustering oriented loss is designed to make the latent representation space more separable. In our experiments, we evaluate our proposed method on three different types of text datasets (namely, TREC-6,Yelp, and DBpedia). Experimental results show that our model achieves the state-of-the-art performance on question, sentiment and topic text datasets. 2 Model Consider a text dataset X with n samples where {xi ∈ X}n …

Content truncated to 500 words