“What Do You Mean by That?” A Parser-Independent Interactive Approach for Enhancing Text-to-SQL

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6913–6922, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6913 “What Do You Mean by That?” A Parser-Independent Interactive Approach for Enhancing Text-to-SQL Yuntao Li1∗, Bei Chen2, Qian Liu3∗, Yan Gao2, Jian-Guang Lou2, Yan Zhang1, Dongmei Zhang2 1Department of Machine Intelligence, Peking University, Beijing, China 2Microsoft Research, Beijing, China; 3Beihang University, Beijing, China 1{li.yt, zhyzhy001}@pku.edu.cn; 3qian.liu@buaa.edu.cn 2{beichen, yan.gao, jlou, dongmeiz}@microsoft.com Abstract In Natural Language Interfaces to Databases systems, the text-to-SQL technique allows users to query databases by using natural lan- guage questions. Though significant progress in this area has been made recently, most parsers may fall short when they are deployed in real systems. One main reason stems from the difficulty of fully understanding the users’ natural language questions. In this paper, we include human in the loop and present a novel parser-independent interactive approach (PIIA) that interacts with users using multi- choice questions and can easily work with arbi- trary parsers. Experiments were conducted on two cross-domain datasets, the WikiSQL and the more complex Spider, with five state-of- the-art parsers. These demonstrated that PIIA is capable of enhancing the text-to-SQL perfor- mance with limited interaction turns by using both simulation and human evaluation. 1 Introduction The past few years have witnessed a burgeoning in- terest in the study of text-to-SQL, the essential tech- nique for Natural Language Interfaces to Databases (NLIDB) systems (Guo et al., 2019; Hwang et al., 2019; He et al., 2019a; Bogin et al., 2019a,b). By converting natural language (NL) questions into executable forms (i.e., Structured Query Language or SQL), text-to-SQL parsers relieve users from the burden of learning about techniques behind the queries. Though significant progress has been made in this field, most parsers are still less than de- sirable when deployed in real NLIDB systems. As users are not experts in database querying, a central challenge for the parsers is to fully understand the users’ NL questions. Since users are who know the questions best, in- teracting with them has been seen as a promising ∗Work done during an internship at Microsoft Research. way to tackle the above challenge in real NLIDB systems. Early works tried to get users involved in checking SQL queries (Li and Jagadish, 2014; Iyer et al., 2017; Yaghmazadeh et al., 2017), which are impracticable in real systems, as they can only succeed if users have a very good knowledge of SQL. In another attempt to involves users, Gur et al. (2018) proposed to interact with non-expert users by multi-choice questions. However, this ap- proach is designed for relatively simple scenarios and cannot be easily applied to more complex ones. More recently, Yao et al. (2019) took an important step forward by measuring uncertainty of neural- based parsers and altering the behavior of them. However, as far as we know, most parsers in real systems are equipped with elaborate rules instead of using fully neural methods (Dhamdhere et al., 2017; Gliozzo et al., 2013; Lai et al., …

Content truncated to 500 words