BERTese: Learning to Speak to BERT

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3618–3623 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3618 BERTese: Learning to Speak to BERT Adi Haviv1 Jonathan Berant1,2 Amir Globerson1 1School of Computer Science, Tel Aviv University 2Allen Institute for AI {adi.haviv,joberant}@cs.tau.ac.il, gamir@post.tau.ac.il Abstract Large pre-trained language models have been shown to encode large amounts of world and commonsense knowledge in their parameters, leading to substantial interest in methods for extracting that knowledge. In past work, knowledge was extracted by taking manually- authored queries and gathering paraphrases for them using a separate pipeline. In this work, we propose a method for automati- cally rewriting queries into “BERTese”, a para- phrase query that is directly optimized towards better knowledge extraction. To encourage meaningful rewrites, we add auxiliary loss functions that encourage the query to corre- spond to actual language tokens. We empiri- cally show our approach outperforms compet- ing baselines, obviating the need for complex pipelines. Moreover, BERTese provides some insight into the type of language that helps lan- guage models perform knowledge extraction. 1 Introduction Recent work has shown that large pre-trained lan- guage models (LM), trained with a masked lan- guage modeling (MLM) objective (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2019; Sanh et al., 2019; Conneau et al., 2020), encode substantial amounts of world knowledge in their parameters. This has led to ample research on developing meth- ods for extracting that knowledge (Petroni et al., 2019, 2020; Jiang et al., 2020; Bouraoui et al., 2020). The most straightforward approach is to present the model with a manually-crafted query such as “Dante was born in [MASK]” and check if the model predicts “Florence” in the [MASK] position. However, when this fails, it is difficult to determine if the knowledge is absent from the LM or if the model failed to understand the query itself. For example, the model might return the correct answer if the query is “Dante was born in the city of [MASK]”. Rewriter (Identity Pretrained BERT) Predictor (Off-The-Shelf Pretrained BERT) BERT Embeddings Vocabulary Q: will & grace was originally aired on [MASK]. Nearest Neighbors Q: will & grace is originally aired on [MASK]. A: nbc Figure 1: The BERTese Model. The model takes an input query, rewrites it, and feeds the output to a pre- trained BERT model. The untrained components are marked in green, and the blue component is trained. Motivated by the above observation, we ask: can we automatically find the best way to “ask” an LM about its knowledge? We refer to this challenge as speaking “BERTese”. In particular, we ask how to rewrite a knowledge-seeking query into one that MLMs understand better, where understanding is manifested by providing a correct answer to the query. Prior work (Jiang et al., 2020) tackled this prob- lem using a 2-step pipeline, where first a small list of paraphrase templates is collected using external resources, and then a model learns to extract …

Content truncated to 500 words