SPE: Symmetrical Prompt Enhancement for Fact Probing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11689 - 11698 December 7-11, 2022 ©2022 Association for Computational Linguistics SPE: Symmetrical Prompt Enhancement for Fact Probing Yiyuan Li∗ UNC-Chapel Hill yiyuanli@cs.unc.edu Tong Che∗† NVIDIA tongc@nvidia.com Yezhen Wang Mila-Quebec AI Institute yezhen.wang@mila.quebec Zhengbao Jiang Carnegie Mellon University zhengbaj@cs.cmu.edu Caiming Xiong Salesforce Research cxiong@salesforce.com Snigdha Chaturvedi UNC-Chapel Hill snigdha@cs.unc.edu Abstract Pretrained language models (PLMs) have been shown to accumulate factual knowledge dur- ing pretraining (Petroni et al., 2019). Recent works probe PLMs for the extent of this knowl- edge through prompts either in discrete or con- tinuous forms. However, these methods do not consider symmetry of the task: object pre- diction and subject prediction. In this work, we propose Symmetrical Prompt Enhancement (SPE), a continuous prompt-based method for factual probing in PLMs that leverages the sym- metry of the task by constructing symmetrical prompts for subject and object prediction. Our results on a popular factual probing dataset, LAMA, show significant improvement of SPE over previous probing methods. 1 Introduction Prompt-based learning proposes to formulate dif- ferent NLP tasks into language modeling problems (Schick and Schütze, 2021). It is a novel paradigm that effectively uses Pretrained Language Models (PLMs) (Liu et al., 2022), and achieves comparable or better performance than fine-tuning (Lester et al., 2021). Prompt-based learning has also been used for the task of factual knowledge probing in PLMs. In this task, the goal is to predict the (masked) ob- ject of factual tuples of type (subject, relation, ob- ject) using PLMs. Prompting methods assume that PLMs gather and store factual knowledge during their pre-training, and cloze-style prompts can be used to probe PLMs to gauge how much knowledge they contain (Petroni et al., 2019). The prompts are either handcrafted (Petroni et al., 2019; Bouraoui et al., 2020) or automatically generated (Shin et al., 2020; Haviv et al., 2021). For example, to probe PLMs about their knowledge of geographic loca- tion of Luxembourg, a prompt can be formed by filling Luxembourg in the first blank of the fol- lowing template: "____ is located in ____.". An ∗ Equal contribution. †Work was done at MILA. Prompt: Luxembourg is located in ____ . Prediction Probability Germany France Europe <Subject: Luxembourg, Relation: location, Object: Europe> Prompt: ____ is located in Germany. Luxembourg Template: ___ is located in ___ . Prompt: ____ is located in France. Luxembourg Prompt: ____ is located in Europe. Luxembourg Germany France Europe Figure 1: Example of factual probing: Given a sub- ject and relation, predict the object. SPE uses a fixed template to generate a prompt for predicting object given subject (green box) as well as several symmet- rical prompts for predicting the subject given object candidates (yellow boxes). The final prediction is ob- tained using the likelihoods of the object candidates and of the given subject as obtained using the symmetrical prompts. Bars represent probabilities from BERT. SPE is a continuous prompt method but we use natural lan- guage prompts and template here for illustration. …

Content truncated to 500 words