Lifelong Learning of Topics and Domain-Specific Word Embeddings

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2294–2309 August 1–6, 2021. ©2021 Association for Computational Linguistics 2294 Lifelong Learning of Topics and Domain-Specific Word Embeddings Xiaorui Qin, Yuyin Lu, Yufu Chen, Yanghui Rao∗ School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China qinxr5@mail2.sysu.edu.cn, luyy37@mail2.sysu.edu.cn, chenyf66@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn Abstract Lifelong topic models mainly focus on in- domain text streams in which each chunk only contains documents from a single domain. To overcome data diversity of the in-domain cor- pus, most of the existing methods exploit the information from limited sources in a separate and heuristic manner. In this study, we develop a lifelong collaborative model (LCM) based on non-negative matrix factorization to accu- rately learn topics and domain-specific word embeddings. LCM particularly investigates: (1) developing a knowledge graph based on the semantic relationships among words in the lifelong learning process, so as to accumulate global context information discovered by topic models and local context information reflected by context word embeddings from previous do- mains, and (2) developing a subword graph based on byte pair encoding and pairwise word relationships to exploit subword information of words in the current in-domain corpus. To the best of our knowledge, we are the first to collaboratively learn topics and word embed- dings via lifelong learning. Experiments on real-world in-domain text streams validate the effectiveness of our method. 1 Introduction Lifelong learning (Silver, 2011; Mitchell et al., 2015), which accumulates and maintains the past knowledge to help future learning in an endless manner, has attracted considerable attention in topic modeling (Chen et al., 2020b; Gupta et al., 2020). Most lifelong topic models (Chen and Liu, 2014b; Chen, 2015; Wang et al., 2016) focus on the corpus that only contains text from a single domain, dubbed the in-domain corpus (Xu et al., 2018). This is because in-domain corpora are widespread in real-world applications, such as breaking news and tweets related to a specific topic (domain). The ∗The corresponding author. key to the success of a lifelong topic model within in-domain corpora is based on a precondition that prior topical information of previous domains can be fully exploited to guide meaningful learning in the new coming domain (Chen and Liu, 2014b). However, because the in-domain corpus is typically of limited size (Xu et al., 2018), it is insufficient for the existing methods to train coherent topics. To alleviate the lack of global context in a cor- pus, one simple solution for topic models is to incorporate general-purpose pre-trained word em- beddings (Das et al., 2015; Xun et al., 2016, 2017b; Dieng et al., 2020). Although the general-purpose embeddings can provide some useful information for words within the in-domain corpus, their em- bedding representations may not be ideal for the target domain and in some cases they may even con- flict with the meanings of the words in the task do- main because words often have multiple senses or meanings (Xu et al., 2018). Another solution trains topics and word embeddings jointly in the one-shot …

Content truncated to 500 words