Supporting Clustering with Contrastive Learning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5419–5430 June 6–11, 2021. ©2021 Association for Computational Linguistics 5419 Supporting Clustering with Contrastive Learning Dejiao Zhang1 Feng Nan1 Xiaokai Wei1 Shang-Wen Li1 Henghui Zhu1 Kathleen McKeown1,2 Ramesh Nallapati1 Andrew O. Arnold1 Bing Xiang1 1AWS AI 2Columbia University dejiaoz,nanfen,xiaokaiw,shangwel,henghui mckeownk,rnallapa,anarnld,bxiang@amazon.com Abstract Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often overlap with each other in the representation space at the beginning of the learning pro- cess, which poses a significant challenge for distance-based clustering in achieving good separation between different categories. To this end, we propose Supporting Clustering with Contrastive Learning (SCCL) – a novel framework to leverage contrastive learning to promote better separation. We assess the per- formance of SCCL on short text clustering and show that SCCL significantly advances the state-of-the-art results on most benchmark datasets with 3%−11% improvement on Accu- racy and 4% − 15% improvement on Normal- ized Mutual Information. Furthermore, our quantitative analysis demonstrates the effec- tiveness of SCCL in leveraging the strengths of both bottom-up instance discrimination and top-down clustering to achieve better intra- cluster and inter-cluster distances when eval- uated with the ground truth cluster labels1. 1 Introduction Clustering, one of the most fundamental challenges in unsupervised learning, has been widely studied for decades. Long established clustering methods such as K-means (MacQueen et al., 1967; Lloyd, 1982) and Gaussian Mixture Models (Celeux and Govaert, 1995) rely on distance measured in the data space, which tends to be ineffective for high- dimensional data. On the other hand, deep neural networks are gaining momentum as an effective way to map data to a low dimensional and hopefully better separable representation space. Many recent research efforts focus on integrat- ing clustering with deep representation learning 1We plan to open source our implementation. Please visit https://arxiv.org/abs/2103.12953 for the release updates. Figure 1: TSNE visualization of the embedding space learned on SearchSnippets using Sentence Transformer (Reimers and Gurevych, 2019a) as backbone. Each color indicates a ground truth semantic category. by optimizing a clustering objective defined in the representation space (Xie et al., 2016; Jiang et al., 2016; Zhang et al., 2017a; Shaham et al., 2018). Despite promising improvements, the clustering performance is still inadequate, especially in the presence of complex data with a large number of clusters. As illustrated in Figure 1, one possible reason is that, even with a deep neural network, data still has significant overlap across categories before clustering starts. Consequently, the clusters learned by optimizing various distance or similarity based clustering objectives suffer from poor purity. On the other hand, Instance-wise Contrastive Learning (Instance-CL) (Wu et al., 2018; Bachman et al., 2019; He et al., 2020; Chen et al., 2020a,b) has recently achieved remarkable success in self- supervised learning. Instance-CL usually optimizes on an auxiliary set obtained by data augmenta- tion. As the name suggests, a contrastive loss is then …

Content truncated to 500 words