Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8938 - 8958 December 7-11, 2022 ©2022 Association for Computational Linguistics RAINIER: Reinforced Knowledge Introspector for Commonsense Question Answering Jiacheng Liu♡ Skyler Hallinan♡ Ximing Lu♡♠ Pengfei He♡ Sean Welleck♡♠ Hannaneh Hajishirzi♡♠ Yejin Choi♡♠ ♡Paul G. Allen School of Computer Science & Engineering, University of Washington ♠Allen Institute for Artificial Intelligence liujc@cs.washington.edu Abstract Knowledge underpins reasoning. Recent re- search demonstrates that when relevant knowl- edge is provided as additional context to com- monsense question answering (QA), it can sub- stantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent. We present RAINIER1, or Reinforced Knowl- edge Introspector, that learns to generate con- textually relevant knowledge in response to given questions. Our approach starts by im- itating knowledge generated by GPT-3, then learns to generate its own knowledge via rein- forcement learning where rewards are shaped based on the increased performance on the re- sulting question answering. RAINIER demon- strates substantial and consistent performance gains when tested over 9 different common- sense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by mod- els that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of com- monsense knowledge elicited from GPT-3. 1 Introduction Commonsense is a significant challenge for mod- ern NLP models, due to the obscurity of underly- ing knowledge that grounds the reasoning process. While humans are generally able to introspect the underlying reasons for their conclusion (Mercier and Sperber, 2017), neural models lack the capabil- ity to verbalize the premises leading to their predic- tion. This hinders models’ performance and robust- ness on commonsense tasks, and makes it difficult 1Code, model and knowledge-extended datasets are avail- able at http://github.com/liujch1998/rainier Figure 1: RAINIER can introspect for commonsense knowledge that underpin the reasoning process, and is trained via reinforcement learning, where the reward is derived from the effectiveness of knowledge when prompting a frozen, generic QA model. to inspect their point of failure. Recent research has demonstrated that relevant knowledge can pro- vide useful context for approaching commonsense tasks. Yet these methods either retrieve from in- domain knowledge bases (Mitra et al., 2019; Chang et al., 2020) that do not have good coverage over commonsense, or generate knowledge from neu- ral models (Shwartz et al., 2020; Gu et al., 2022; Liu et al., 2022), which often need domain-specific engineering and very large models (e.g. GPT-3 (Brown et al., 2020)). It is still an open challenge to systematically find high-quality knowledge. In this work, we use a novel, reinforcement- learning-based method to develop RAINIER, a gen- erative neural model that can introspect …

Content truncated to 500 words