Semantic Parsing to Probabilistic Programs for Situated Question Answering

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 160–170, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Semantic Parsing to Probabilistic Programs for Situated Question Answering Jayant Krishnamurthy and Oyvind Tafjord and Aniruddha Kembhavi Allen Institute for Artificial Intelligence jayantk,oyvindt,anik@allenai.org Abstract Situated question answering is the problem of answering questions about an environment such as an image or diagram. This problem requires jointly interpreting a question and an environment using background knowledge to select the correct answer. We present Parsing to Probabilistic Programs (P 3), a novel situ- ated question answering model that can use background knowledge and global features of the question/environment interpretation while retaining efficient approximate inference. Our key insight is to treat semantic parses as prob- abilistic programs that execute nondetermin- istically and whose possible executions repre- sent environmental uncertainty. We evaluate our approach on a new, publicly-released data set of 5000 science diagram questions, outper- forming several competitive classical and neu- ral baselines. 1 Introduction Situated question answering is a challenging prob- lem that requires reasoning about uncertain inter- pretations of both a question and an environment together with background knowledge to determine the answer. To illustrate these challenges, consider the 8th grade science diagram questions in Figure 1, which are motivated by the Aristo project (Clark and Etzioni, 2016). These questions require both com- puter vision to interpret the diagram and composi- tional question understanding. These components, being imperfect, introduce uncertainty that must be jointly reasoned about to avoid implausible interpre- tations. These uncertain interpretations must further 1. According to the given food chain, what is the num- ber of organisms that eat deer? (A) 3 (B) 2 (C) 4 (D) 1 2. Which organism is both predator and prey? (A) Bark Beetles (B) Insect-eating birds (C) Deer (D) Hawks 3. Based on the given food web, what would happen if there were no insect-eating birds? (A) The grasshop- per population would increase. (B) The grasshop- per population would decrease. (C) There would be no change in grasshopper number. Figure 1: Example food web questions. A food web depicts a collection of organisms in an ecosystem with an arrow from organism x to y indicating that y eats x. Questions may require counting (1), knowing animal roles (2) and reasoning about population changes (3). be combined with background knowledge, such as the definition of a “predator,” to determine the cor- rect answer. The challenges of situated question answering have not been completely addressed by prior work. Early “possible worlds” models (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013; Mali- nowski and Fritz, 2014) were capable of composi- tional question understanding and using background knowledge, but did not jointly reason about environ- 160 ment/question uncertainty. These models also used unscalable inference algorithms for reasoning about the environment, despite the lack of joint reasoning. More recent neural models (Antol et al., 2015; Mali- nowski et al., 2015; Yang et al., 2015) are incapable of using background knowledge and it remains …

Content truncated to 500 words