DREAM: Improving Situational QA by First Elaborating the Situation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115 - 1127 July 10-15, 2022 ©2022 Association for Computational Linguistics DREAM: Improving Situational QA by First Elaborating the Situation Yuling Gu∗, Bhavana Dalvi Mishra, Peter Clark Allen Institute for AI, Seattle, WA {yulingg,bhavanad,peterc}@allenai.org Abstract When people answer questions about a spe- cific situation, e.g., “I cheated on my mid-term exam last week. Was that wrong?”, cognitive science suggests that they form a mental pic- ture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the ques- tion situation, elaborating the “scene”. To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then pro- vide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elab- orations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context im- proves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM’s training data. These results suggest that adding focused elaborations about a situa- tion can improve a system’s reasoning about it, and may serve as an effective way of injecting new scenario-based knowledge into QA mod- els. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters.1 1 Introduction Cognitive science has long promoted the forma- tion of mental models - coherent, constructed rep- resentations of the situations we encounter - as central to understanding and question-answering (Johnson-Laird, 1983). Drawing loosely on this ∗First author was a student at University of Washington at the time of submission of this paper. This is work done during an internship at Allen Institute for AI. 1We make our dataset and model publicly available at https://github.com/allenai/dream. Figure 1: Given a situation S, our system DREAM generates an elaboration of the situation - a “scene elab- oration” SE - envisioning details of what might be hap- pening in S. Given a question Q about S, we find that a SOTA QA system (Macaw) answers Q more accurately when SE is provided as additional input. idea, but without making any claims of how lan- guage models (LM) reason internally, our goal is to investigate if a LM can answer situated questions about social situations more accurately if they are provided with additional, pertinent details about those situations before answering. To explore this goal, we first investigate how well a representative state-of-the-art language model can generate such scene elaborations zero-shot, by asking it probing questions. We find that such zero-shot elaborations are generally poor quality. To overcome this, we train a new model, called DREAM,2 …

Content truncated to 500 words