Patton: Language Model Pretraining on Text-Rich Networks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7005–7020 July 9-14, 2023 ©2023 Association for Computational Linguistics PATTON : Language Model Pretraining on Text-Rich Networks Bowen Jin1, Wentao Zhang1, Yu Zhang1, Yu Meng1, Xinyang Zhang1, Qi Zhu1, Jiawei Han1 1University of Illinois at Urbana-Champaign, IL, USA {bowenj4,wentao4,yuz9,yumeng5,xz43,qiz3,hanj}@illinois.edu Abstract A real-world text corpus sometimes comprises not only text documents, but also semantic links between them (e.g., academic papers in a bibliographic network are linked by citations and co-authorships). Text documents and se- mantic connections form a text-rich network, which empowers a wide range of downstream tasks such as classification and retrieval. How- ever, pretraining methods for such structures are still lacking, making it difficult to build one generic model that can be adapted to various tasks on text-rich networks. Current pretrain- ing objectives, such as masked language mod- eling, purely model texts and do not take inter- document structure information into considera- tion. To this end, we propose our PretrAining on TexT-Rich NetwOrk framework PATTON. PATTON1 includes two pretraining strategies: network-contextualized masked language mod- eling and masked node prediction, to capture the inherent dependency between textual at- tributes and network structure. We conduct experiments on four downstream tasks in five datasets from both academic and e-commerce domains, where PATTON outperforms baselines significantly and consistently. 1 Introduction Texts in the real world are often interconnected through links that can indicate their semantic relationships. For example, papers connected through citation links tend to be of similar topics; e-commerce items connected through co-viewed links usually have related functions. The texts and links together form a type of network called a text- rich network, where documents are represented as nodes, and the edges reflect the links among documents. Given a text-rich network, people are usually interested in various downstream tasks (e.g., document/node classification, document retrieval, and link prediction) (Zhang et al., 2019; Wang et al., 1Code is available at https://github.com/ PeterGriffinJin/Patton 2019; Jin et al., 2023a). For example, given a com- puter science academic network as context, it is intuitively appealing to automatically classify each paper (Kandimalla et al., 2021), find the authors of a new paper (Schulz et al., 2014), and provide pa- per recommendations (Küçüktunç et al., 2012). In such cases, pretraining a language model on a given text-rich network which can benefit a great number of downstream tasks inside this given network is highly demanded (Hu et al., 2020b). While there have been abundant studies on build- ing generic pretrained language models (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020), they are mostly designed for modeling texts exclusively, and do not consider inter-document structures. Along another line of research, various network-based pretraining strate- gies are proposed in the graph learning domain to take into account structure information (Hu et al., 2020a,b). Yet, they focus on pretraining graph neural networks rather than language models and cannot easily model the rich textual semantic in- formation in the …

Content truncated to 500 words