Evaluating Semantic Parsing against a Simple Web-based Question Answering Model

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 161–167, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Evaluating Semantic Parsing against a Simple Web-based Question Answering Model Alon Talmor Tel-Aviv University alontalmor@mail.tau.ac.il Mor Geva Tel-Aviv University morgeva@mail.tau.ac.il Jonathan Berant Tel-Aviv University joberant@cs.tau.ac.il Abstract Semantic parsing shines at analyzing com- plex natural language that involves com- position and computation over multiple pieces of evidence. However, datasets for semantic parsing contain many factoid questions that can be answered from a sin- gle web document. In this paper, we pro- pose to evaluate semantic parsing-based question answering models by comparing them to a question answering baseline that queries the web and extracts the answer only from web snippets, without access to the target knowledge-base. We investigate this approach on COMPLEXQUESTIONS, a dataset designed to focus on composi- tional language, and find that our model obtains reasonable performance (∼35 F1 compared to 41 F1 of state-of-the-art). We find in our analysis that our model per- forms well on complex questions involv- ing conjunctions, but struggles on ques- tions that involve relation composition and superlatives. 1 Introduction Question answering (QA) has witnessed a surge of interest in recent years (Hill et al., 2015; Yang et al., 2015; Pasupat and Liang, 2015; Chen et al., 2016; Joshi et al., 2017), as it is one of the promi- nent tests for natural language understanding. QA can be coarsely divided into semantic parsing- based QA, where a question is translated into a logical form that is executed against a knowledge- base (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Liang et al., 2011; Kwiatkowski et al., 2013; Reddy et al., 2014; Berant and Liang, 2015), and unstructured QA, where a ques- tion is answered directly from some relevant text (Voorhees and Tice, 2000; Hermann et al., 2015; Hewlett et al., 2016; Kadlec et al., 2016; Seo et al., 2016). In semantic parsing, background knowledge has already been compiled into a knowledge-base (KB), and thus the challenge is in interpreting the question, which may contain compositional con- structions (“What is the second-highest mountain in Europe?”) or computations (“What is the dif- ference in population between France and Ger- many?”). In unstructured QA, the model needs to also interpret the language of a document, and thus most datasets focus on matching the question against the document and extracting the answer from some local context, such as a sentence or a paragraph (Onishi et al., 2016; Rajpurkar et al., 2016; Yang et al., 2015). Since semantic parsing models excel at han- dling complex linguistic constructions and reason- ing over multiple facts, a natural way to exam- ine whether a benchmark indeed requires model- ing these properties, is to train an unstructured QA model, and check if it under-performs compared to semantic parsing models. If questions can be answered by examining local contexts only, then the use of a knowledge-base is perhaps unneces- sary. However, to the best of our knowledge, only models that …

Content truncated to 500 words