Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2065–2077 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2065 Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback Ahmed Elgohary∗ University of Maryland, College Park elgohary@cs.umd.edu Saghar Hosseini, Ahmed Hassan Awadallah Microsoft Research, Redmond, WA {sahoss,hassanam}@microsoft.com Abstract We study the task of semantic parse correction with natural language feedback. Given a natu- ral language utterance, most semantic parsing systems pose the problem as one-shot transla- tion where the utterance is mapped to a cor- responding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feed- back to correct the system when it generates an inaccurate interpretation of an initial utter- ance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of ut- terances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the cor- rection task and show that incorporating such a rich form of feedback can significantly im- prove the overall semantic parsing accuracy while retaining the flexibility of natural lan- guage interaction. While we estimated hu- man correction accuracy is 81.5%, our best model achieves only 25.1%, which leaves a large gap for improvement in future research. SPLASH is publicly available at https:// aka.ms/Splash_dataset. 1 Introduction Natural language interfaces (NLIs) have been the “holy grail" of natural language understating and human-computer interaction for decades (Woods et al., 1972; Codd, 1974; Hendrix et al., 1978; Zettlemoyer and Collins, 2005). However, early attempts in building NLIs to databases did not achieve the expected success due to limitations in language understanding capability, among other reasons (Androutsopoulos et al., 1995; Jones and Galliers, 1995). NLIs have been receiving increas- ing attention recently motivated by interest in de- veloping virtual assistants, dialogue systems, and ∗Most work was done while the first author was an intern at Microsoft Research. Find all the locations whose names contain the word "film" Address 770 Edd Lane Apt. 098 14034 Kohler Drive finding the Address of Locations table for which Location_Name contains "film" Address is wrong. I want the name of the locations Location_Name Film Festival Film Castle finding the Location_Name of Locations table for which Location_Name contains "film" … … Figure 1: An example of human interaction with a Text- to-SQL system to correct the interpretation of an input utterance. The system generates an initial SQL parse, explains it in natural language, and displays the execu- tion result. Then, the system uses the human-provided natural language feedback to correct the initial parse. semantic parsing systems. NLIs to databases were at the forefront of this wave with several studies fo- cusing on parsing natural language utterances into an executable SQL queries (Text-to-SQL parsing). Most of the work addressing the Text-to-SQL problem (and semantic parsing in general) frames it as a one-shot mapping problem. We establish (Sec- tion 4.1) that the majority of parsing mistakes that recent …

Content truncated to 500 words