UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6159 - 6169 May 22-27, 2022 c⃝2022 Association for Computational Linguistics UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining Jiacheng Li, Jingbo Shang, Julian McAuley University of California, San Diego {j9li,jshang,jmcauley}@eng.ucsd.edu Abstract High-quality phrase representations are essen- tial to finding topics and related terms in doc- uments (a.k.a. topic mining). Existing phrase representation learning methods either simply combine unigram representations in a context- free manner or rely on extensive annotations to learn context-aware knowledge. In this paper, we propose UCTOPIC, a novel unsu- pervised contrastive learning framework for context-aware phrase representations and topic mining. UCTOPIC is pretrained in a large scale to distinguish if the contexts of two phrase mentions have the same semantics. The key to pretraining is positive pair construction from our phrase-oriented assumptions. How- ever, we find traditional in-batch negatives cause performance decay when finetuning on a dataset with small topic numbers. Hence, we propose cluster-assisted contrastive learning (CCL) which largely reduces noisy negatives by selecting negatives from clusters and fur- ther improves phrase representations for top- ics accordingly. UCTOPIC outperforms the state-of-the-art phrase representation model by 38.2% NMI in average on four entity cluster- ing tasks. Comprehensive evaluation on topic mining shows that UCTOPIC can extract co- herent and diverse topical phrases. 1 Introduction Topic modeling discovers abstract ’topics’ in a collection of documents. A topic is typically modeled as a distribution over terms. High- quality phrase representations help topic mod- els understand phrase semantics in order to find well-separated topics and extract coherent phrases. Some phrase representation methods (Wang et al., 2021; Yu and Dredze, 2015; Zhou et al., 2017) learn context-free representations by unigram em- bedding combination. Context-free representations tend to extract similar phrases mentions (e.g. “great food” and “good food”, see Section 4.3). Context- aware methods such as DensePhrase (Lee et al., The United States is a federation of 50 individual states. Irving Washington’s book was popular in the United States. The [MASK] [MASK] is a federation of 50 individual states. Irving Washington’s book was popular in the [MASK] [MASK]. ①: The semantics of phrases are determined by their context. ②: Phrases that have the same mentions have the same semantics. Positive pairs (same semantics) ① ② Figure 1: Two assumptions used in UCTOPIC to pro- duce positive pairs for contrastive learning. 2021) and LUKE (Yamada et al., 2020) need super- vision from task-specific datasets or distant anno- tations with knowledge bases. Manual or distant supervision limits the ability to represent out-of- vocabulary phrases especially for domain-specific datasets. Recently, contrastive learning has shown effectiveness for unsupervised representation learn- ing in visual (Chen et al., 2020) and textual (Gao et al., 2021) domains. In this work, we seek to advance state-of-the- art phrase representation methods and demonstrate that a contrastive objective can be extremely effec- tive at learning phrase semantics in sentences. We present UCTOPIC, an Unsupervised Contrastive learning framework for phrase representations and TOPIC …

Content truncated to 500 words