Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 1733 - 1745 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models Simran Arora Sen Wu Enci Liu Christopher Ré Department of Computer Science, Stanford University {simran, senwu, jesslec, chrismre}@cs.stanford.edu Abstract Popular language models (LMs) struggle to capture knowledge about rare tail facts and entities. Since widely used systems such as search and personal-assistants must support the long tail of entities that users ask about, there has been significant effort towards en- hancing these base LMs with factual knowl- edge. We observe proposed methods typically start with a base LM and data that has been annotated with entity metadata, then change the model, by modifying the architecture or introducing auxiliary loss terms to better cap- ture entity knowledge. In this work, we ques- tion this typical process and ask to what ex- tent can we match the quality of model mod- ifications, with a simple alternative: using a base LM and only changing the data. We propose metadata shaping, a method which inserts substrings corresponding to the read- ily available entity metadata, e.g. types and descriptions, into examples at train and infer- ence time based on mutual information. De- spite its simplicity, metadata shaping is quite effective. On standard evaluation benchmarks for knowledge-enhanced LMs, the method ex- ceeds the base-LM baseline by an average of 4.3 F1 points and achieves state-of-the-art re- sults. We further show the gains are on average 4.4x larger for the slice of examples containing tail vs. popular entities. 1 Introduction Recent language models (LMs) such as BERT (De- vlin et al., 2019) and its successors are remark- able at memorizing knowledge seen frequently dur- ing training, however performance degrades over the long tail of rare facts. Given the importance of factual knowledge for tasks such as question- answering, search, and personal assistants (Bern- stein et al., 2012; Poerner et al., 2020; Orr et al., 2020), there has been significant interest in inject- ing these base LMs with factual knowledge about entities (Zhang et al., 2019; Peters et al., 2019, inter alia.). In this work, we work we propose a simple and effective approach for enhancing LMs with knowledge, called metadata shaping. Existing methods to capture entity knowledge more reliably, typically use the following steps: first annotating natural language text with entity metadata, and next modifying the base LM model to learn from the tagged data. Entity metadata is obtained by linking substrings of text to entries in a knowledge base such as Wikidata, which stores en- tity IDs, types, descriptions, and relations. Model modifications include introducing continuous vec- tor representations for entities or auxiliary objec- tives (Zhang et al., 2019; Peters et al., 2019; Ya- mada et al., 2020; Wang et al., 2020; Xiong et al., 2020; Joshi et al., 2020a; Su et al., 2021). Other methods combine multiple learned modules, which are each specialized to handle fine-grained rea- soning patterns or subsets of the data distribution …

Content truncated to 500 words