Generated Knowledge Prompting for Commonsense Reasoning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3154 - 3169 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Generated Knowledge Prompting for Commonsense Reasoning Jiacheng Liu♥ Alisa Liu♥ Ximing Lu♥♠ Sean Welleck♥♠ Peter West♥♠ Ronan Le Bras♠ Yejin Choi♥♠ Hannaneh Hajishirzi♥♠ ♥Paul G. Allen School of Computer Science & Engineering, University of Washington ♠Allen Institute for Artificial Intelligence liujc@cs.washington.edu Abstract It remains an open question whether incorpo- rating external knowledge benefits common- sense reasoning while maintaining the flexi- bility of pretrained sequence models. To in- vestigate this question, we develop generated knowledge prompting, which consists of gen- erating knowledge from a language model, then providing the knowledge as additional in- put when answering a question. Our method does not require task-specific supervision for knowledge integration, or access to a struc- tured knowledge base, yet it improves perfor- mance of large-scale, state-of-the-art models on four commonsense reasoning tasks, achiev- ing state-of-the-art results on numerical com- monsense (NumerSense), general common- sense (CommonsenseQA 2.0), and scientific commonsense (QASC) benchmarks. Gener- ated knowledge prompting highlights large- scale language models as flexible sources of external knowledge for improving common- sense reasoning. Our code is available at github.com/liujch1998/GKP 1 Introduction It remains an open research question whether exter- nal knowledge is needed for commonsense reason- ing. On one hand, a substantial body of prior work has reported that integrating external knowledge can help improve task performance (Mitra et al., 2019; Bian et al., 2021, inter alia), especially if the knowledge is high quality (e.g. hand-crafted by ex- perts). On the other hand, recent leaderboards are often dominated by large-scale pretrained models that are fine-tuned on a target benchmark (Khashabi et al., 2020; Lourie et al., 2021), suggesting that the benefits of external knowledge may wash away as the underlying models increase in size and are pretrained on ever larger amounts of raw text. Even if external knowledge is found to be ef- fective on a particular task, flexibility remains a fundamental hurdle to integrating external knowl- Figure 1: Generated knowledge prompting involves (i) using few-shot demonstrations to generate question- related knowledge statements from a language model; (ii) using a second language model to make predic- tions with each knowledge statement, then selecting the highest-confidence prediction. edge, as many benchmarks currently lack appropri- ate knowledge bases with sufficient coverage. Fur- thermore, prior methods often require task-specific, custom supervision for knowledge integration (Mi- tra et al., 2019; Chang et al., 2020), introducing a burden for rapidly adapting new pretrained models to a wide variety of tasks. In this paper, we investigate whether external knowledge can be helpful for commonsense rea- soning, even on top of the largest state-of-the-art pretrained models (e.g. T5-11b (Raffel et al., 2019) and its variants), with a focus on four recent com- monsense benchmarks. To facilitate easier adap- tation with any zero-shot or finetuned models, we propose an approach that does not require access to a structured knowledge base or joint finetuning for knowledge …

Content truncated to 500 words