Dataset for a Neural Natural Language Interface for Databases (NNLIDB)

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 906–914, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Dataset for a Neural Natural Language Interface for Databases (NNLIDB) Florin Brad1, Radu Iacob2, Ionel Hosu2, and Traian Rebedea2 1Bitdefender Romania fbrad@bitdefender.com 2University Politehnica of Bucharest {radu.iacob23, ionel.hosu, traian.rebedea}@gmail.com Abstract Progress in natural language interfaces to databases (NLIDB) has been slow mainly due to linguistic issues (such as language ambiguity) and domain portability. More- over, the lack of a large corpus to be used as a standard benchmark has made data- driven approaches difficult to develop and compare. In this paper, we revisit the problem of NLIDBs and recast it as a se- quence translation problem. To this end, we introduce a large dataset extracted from the Stack Exchange Data Explorer web- site, which can be used for training neural natural language interfaces for databases. We also report encouraging baseline re- sults on a smaller manually annotated test corpus, obtained using an attention-based sequence-to-sequence neural network. 1 Introduction Natural language interfaces have received atten- tion as tools for simplifying the interaction be- tween users and computers. These interfaces often exclude or complement input devices, such as key- board or touch screens, or even specific languages used for interacting with an application. A more focused area is composed of Natural Language In- terface to Databases (NLIDB), which would allow a person to retrieve useful information from any database without knowledge of specific query lan- guages such as structured query language (SQL) for relational databases. Despite initial efforts into NLIDBs started decades ago, research has advanced slowly and at this moment there are no commercial solutions or widespread prototypes. The main difficulties in solving this problem stem from linguistic failures and the inability to develop general-purpose solu- tions that are portable to different databases and schemas. Due to the recent success of deep neural ap- proaches in natural language processing, our aim is twofold. First, we hope to rejuvenate interest in the NLIDB problem by proposing a large dataset, called the Stack Exchange Natural Language In- terface to Database (SENLIDB) corpus, for de- veloping data-driven machine learning models and for reporting progress. The training set consists of 24, 890 pairs (textual description, SQL snippet) crawled using the Stack Exchange API that we fil- tered and cleaned. A smaller test set consisting of 780 pairs that were manually created by two anno- tators is also available for comparing solutions. Second, we report results on a neural base- line that uses an attention-enhanced sequence- to-sequence (SEQ2SEQ) architecture (Bahdanau et al., 2014) to model the conditional probability of an SQL query given a natural language descrip- tion. This model is trained on the aforementioned dataset and its performance is computed both us- ing cross-validation and on the manually labeled test set. Qualitative results reveal code that is syn- tactically correct most of the times and closely re- lated to the user’s intention. Moreover, we report results on two smaller tasks, which …

Content truncated to 500 words