KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2116–2138 July 9-14, 2023 ©2023 Association for Computational Linguistics KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding Shangbin Feng1 Zhaoxuan Tan2 Wenqian Zhang2 Zhenyu Lei2 Yulia Tsvetkov1 1University of Washington 2Xi’an Jiaotong University {shangbin, yuliats}@cs.washington.edu {tanzhaoxuan, 2194510944, fischer}@stu.xjtu.edu.cn Abstract With the advent of pretrained language mod- els (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks. These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pretrained LMs. While existing ap- proaches have leveraged external knowledge, it remains an open question how to jointly in- corporate knowledge graphs representing vary- ing contexts—from local (e.g., sentence), to document-level, to global knowledge—to en- able knowledge-rich exchange across these con- texts. Such rich contextualization can be es- pecially beneficial for long document under- standing tasks since standard pretrained LMs are typically bounded by the input sequence length. In light of these challenges, we pro- pose KALM, a Knowledge-Aware Language Model that jointly leverages knowledge in lo- cal, document-level, and global contexts for long document understanding. KALM first en- codes long documents and knowledge graphs into the three knowledge-aware context repre- sentations. It then processes each context with context-specific layers, followed by a “con- text fusion” layer that facilitates knowledge exchange to derive an overarching document representation. Extensive experiments demon- strate that KALM achieves state-of-the-art per- formance on six long document understanding tasks and datasets. Further analyses reveal that the three knowledge-aware contexts are com- plementary and they all contribute to model performance, while the importance and infor- mation exchange patterns of different contexts vary with respect to different tasks and datasets. 1 1Code and data are publicly available at https://github. com/BunsenFeng/KALM. 1 Introduction Large language models (LMs) have become the dominant paradigm in NLP research, while knowl- edge graphs (KGs) are the de facto standard of symbolic knowledge representation. Recent ad- vances in knowledge-aware NLP focus on combin- ing the two paradigms (Wang et al., 2021b; Zhang et al., 2021; He et al., 2021), infusing encyclopedic (Vrandeˇci´c and Krötzsch, 2014; Pellissier Tanon et al., 2020), commonsense (Speer et al., 2017), and domain-specific (Feng et al., 2021; Chang et al., 2020) knowledge with LMs. Knowledge- grounded models achieved state-of-the-art perfor- mance in tasks including question answering (Sun et al., 2022), commonsense reasoning (Kim et al., 2022; Liu et al., 2021), and social text analysis (Zhang et al., 2022; Hu et al., 2021). Prior approaches to infusing LMs with knowl- edge typically focused on three hitherto orthogonal directions: incorporating knowledge related to lo- cal (e.g., sentence-level), document-level, or global context. Local context approaches argue that sen- tences mention entities, and the external knowledge of entities, such as textual descriptions (Balachan- dran et al., 2021; Wang et al., 2021b) and metadata (Ostapenko et al., 2022), help LMs realize they are more than tokens. Document-level approaches ar- gue …

Content truncated to 500 words