LinkBERT: Pretraining Language Models with Document Links

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8003 - 8016 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LinkBERT: Pretraining Language Models with Document Links Michihiro Yasunaga Jure Leskovec∗ Percy Liang∗ Stanford University {myasu,jure,pliang}@cs.stanford.edu Abstract Language model (LM) pretraining captures vari- ous knowledge from text corpora, helping down- stream NLP tasks. However, existing methods such as BERT model a single document, failing to capture document dependencies and knowl- edge that spans across documents. In this work, we propose LinkBERT, an effective LM pretrain- ing method that incorporates document links, such as hyperlinks. Given a pretraining corpus, we view it as a graph of documents, and create LM inputs by placing linked documents in the same context. We then train the LM with two joint self-supervised tasks: masked language modeling and our newly proposed task, docu- ment relation prediction. We study LinkBERT in two domains: general domain (pretrained on Wikipedia with hyperlinks) and biomedical do- main (pretrained on PubMed with citation links). LinkBERT outperforms BERT on various down- stream tasks in both domains. It is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT attains new state-of-the-art on various BioNLP tasks (+7% on BioASQ and USMLE). We release the pretrained models, LinkBERT and BioLinkBERT, as well as code and data.1 1 Introduction Pretrained language models (LMs), like BERT and GPTs (Devlin et al., 2019; Brown et al., 2020), have shown remarkable performance on many natural language processing (NLP) tasks, such as text classification and question answering (Raffel et al., 2020), becoming the foundation of modern NLP systems. By performing self-supervised learning on text, such as masked language modeling (Devlin et al., 2019), LMs learn to encode various knowl- edge from text corpora and produce informative language representations for downstream tasks (Bosselut et al., 2019; Bommasani et al., 2021). ∗ Equal senior authorship. 1Available at https://github.com/michiyasunaga/ LinkBERT. Languag [CLS] The Tidal Basin [SE ... Segment A Mask (DRP) ● Linked ● Random ● Contiguous Corpus with Document Links (e.g. hyperlink, reference) Doc 2 Doc 5 Doc 4 Doc 6 Doc 1 Doc 3 Create LM inputs Train LMs Segment A Segment B Doc 1.1 Doc 1.2 Doc 1.1 Doc 5.2 Doc 1.1 Doc 3.1 Contiguous Random Linked or or [Tidal Basin, Washington D.C.] The Tidal Basin is a man-made reservoir located between the Potomac River and the Washington Channel in Washington, D.C. It is part of West Potomac Park, is near the National Mall and is a focal point of the National Cherry Blossom Festival held each spring. The Jefferson Memorial, the Martin Luther King Jr. Memorial, the Franklin Delano Roosevelt Memorial, and the George Mason Memorial are situated adjacent to the Tidal Basin. [The National Cherry Blossom Festival] ... It is a spring celebration commemorating the March 27, 1912, gift of Japanese cherry trees from Mayor of Tokyo City Yukio Ozaki to the city of Washington, D.C. Mayor Ozaki gifted the …

Content truncated to 500 words