Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability

Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability Ana-Maria Popescu Alex Armanasu Oren Etzioni University of Washington {amp, alexarm, etzioni, daveko, ayates}@cs.washington.edu David Ko Alexander Yates Abstract Natural Language Interfaces to Databases (NLIs) can benefit from the advances in statis- tical parsing over the last fifteen years or so. However, statistical parsers require training on a massive, labeled corpus, and manually cre- ating such a corpus for each database is pro- hibitively expensive. To address this quandary, this paper reports on the PRECISE NLI, which uses a statistical parser as a “plug in”. The pa- per shows how a strong semantic model cou- pled with “light re-training” enables PRECISE to overcome parser errors, and correctly map from parsed questions to the corresponding SQL queries. We discuss the issues in using statistical parsers to build database-independent NLIs, and report on experimental results with the benchmark ATIS data set where PRECISE achieves 94% accuracy. 1 Introduction and Motivation Over the last fi fteen years or so, much of the NLP community has focused on the use of statistical and machine learning techniques to solve a wide range of problems in parsing, machine translation, and more. Yet, classical problems such as building Natural Language Interfaces to Databases (NLIs) (Grosz et al., 1987) are far from solved. There are many reasons for the limited success of past NLI efforts (Androutsopoulos et al., 1995). We highlight several problems that are remedied by our approach. First, manually authoring and tuning a se- mantic grammar for each new database is brittle and prohibitively expensive. In response, we have im- plemented a “transportable” NLI that aims to mini- mize manual, database-specifi c confi guration. Sec- ond, NLI systems built in the 70s and 80s had lim- ited syntactic parsing capabilities. Thus, we have an opportunity to incorporate the important advances made by statistical parsers over the last two decades in an NLI. However, attempting to use a statistical parser in a database-independent NLI leads to a quandary. On the one hand, to parse questions posed to a particu- lar database, the parser has to be trained on a corpus of questions specifi c to that database. Otherwise, many of the parser’s decisions will be incorrect. For example, the Charniak parser (trained on the 40,000 sentences in the WSJ portion of the Penn Treebank) treats ‘list’ as a noun, but in the context of the ATIS database it is a verb.1 On the other hand, manually creating and labeling a massive corpus of questions for each database is prohibitively expensive. We consider two methods of resolving this quandary and assess their performance individually and in concert on the ATIS data set. First, we use a strong semantic model to correct parsing errors. We introduce a theoretical framework for discrim- inating between Semantically Tractable (ST) ques- tions and diffi cult ones, and we show that ST ques- tions are prevalent in the well-studied ATIS data set (Price, 1990). Thus, we show that the seman- tic component …

Content truncated to 500 words